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(57) Abstract: A system and method for autofusion of three-dimensional ("3-D") image data volumes is described. According 
to the system and method, registration is effected by simultaneously displaying on a GUI at least, but preferably, two 3-D, image 
data volumes, and one of the 3-D image data volumes is held fixed while the other may be scaled, rotated, and translated to align 
homologous anatomic features. The system and method will simultaneously display the axial, sagittal, and coronal plane views of a 
region of a region of interest. These plane views will be of data volumes that are designed and configured for a system user to "drag" 
or "position" one volume to a new position in one plane view which will also simultaneously update the data volume in the other two 
plane views with no apparent lag time. The system and method also includes automated alignment computation based on mutual 
information (MI) maximization. The system and method aligns 3-D more efficiently and faster than prior image alignment methods. 
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Automated Image Fusion/ 
Alignment System and 
Method 

Field of the Invention 

The present invention relates to systems and methods for aligning 
image data volumes. More specifically, the present invention relates to aligning three- 
dimensional image data volumes. 

Background of the Invention 

In the medical treatment area, it is usually necessary to utilize a number 
of modalities to view the internal anatomy of the person to be treated. These modalities 
include X-ray imaging, Magnetic Resonance Imaging ("MRI"), and computed tomog- 
raphy ("CT') imaging. There also are other modalities such as functional MRI 
O'fMRT), single photon emission computed tomography ("SPECT"), and positron 
emission tomography ("PET'), all of whose images contain physiologic or metabolic 
information depicting the actions of living tissue. 

It is known that each of modalities has certain strengths and weaknesses 
in the images that it produces. For example, X-radiography ("X-ray") imaging has high 
spatial and intensity resolutions, shows bony anatomy with high detail, and is rela- 
tively inexpensive to use; but X-ray also presents the viewer with complex two-dimen- 
sional ("2-D") views of superimposed anatomy. X-radiography also has difficulty 
resolving soft tissue features. MRI has the advantage of displaying three-dimensional 
("3-D") images of soft tissues with high contrast and high spatial resolution, but does 
not image bone well. CT imagery, based on X-ray absorption, produces 3-D pictures of 
bony anatomy, and, increasingly, good definition of soft tissue, although MRI remains 
the preferred modality for viewing soft tissue. However, if the correct modalities are 
selected and aligned, the resulting combined image data will provide a more complete 
representation of the internal anatomy of the patient. Given this, the issue is to identify 
a system and method that will align image data from two modalities very accurately in 
a reasonable amount of time. 
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Image alignment, which is part of the science of image fusion, has been 
used since at least World War II when time course aerial photography was used to sup- 
port strategic bombing. Blink comparator machines were used to superimpose two 
images taken at different times by displaying them to the viewer in rapid succession. 
5 As these images were being displayed in this manner, their features were matched to 
quantify the bombing damage. 

This same basic comparison technique has been used to superimpose 
images produced by remote sensors, for example aerial or satellite sensors, but with the 
refinement of using computer-based methods to align images produced at different 

10 times, or at the same time by different channels of the sensor. These images were digi- 
tal images. The techniques just described for image alignment in remote sensors were 
reported in Castleman, K.R., "Fundamentals of Digital Picture Processing," Prentice- 
Hall , Englewood Cliffs, New Jersey, 1979, and Moik, J.G., "Digital Processing of 
Remotely Sensed Images;' NASA SP-43L Washington, D.C., 1980. 

15 The use of computers to effect image alignment became a necessity as 

the number of images produced by LANDS AT, SPOT, and other satellite systems rap- 
idly grew and there was the need to perform nonlinear warping transforms to match the 
images taken from different sky-to-ground perspectives. The use of computers also 
became a necessity to effect alignment of brain tissue images so that useful and effec- 

20 tive neurophysiological research could be conducted. This was reported in Hibbard, L, 
etal, Science , 236:1641-1646, 1987. Reviews of various computed alignment tech- 
niques have been reported in Brown, L.G., "A Survey of Image Registration Tech- 
niques," ACM Compu tin g Surveys . 24: 325-376, 1992 and Van den Elsen, P.A., et al, 
"Medical Image Matching - A Review with Classification," IEEE Engineering in 

25 Medicine and Biology , March 1993, pp. 26-39. 

Image fusion of images from at least two modalities is currently being 
used in radiation onocology because it has been found to provide better tumor defini- 
tion, which was reported in Rosenman, J.G., et aL, "Image Registration: An Essential 
Part of Radiation Therapy Treatment Planning," International Journal of Radiation 

30 Onocology. P mln ov, and Phvsics , 40: 197-205, 1998. It is anticipated that there will be 
increased development of software tools for the use in radiation treatment planning 
("RTP") to effect image alignment and display/contouring tools to process and use the 
fused images that are produced. 
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Image alignment methods generally fall in two categories: manual 
alignment by direct inspection of the images being aligned, and automatic alignment 
by computing a solution to a numerical problem with some type of computer program. 
Manual alignment ("MA") is carried out visually by matching corresponding features 

5 of two images. MA may be implemented in all RTP systems that offer or purport to 
offer image fusion. The mechanism that most frequently is used in MA is the visual 
placement of fiducial points or markers from which a transformation is derived, for 
example, by a least-square minimization of the differences between corresponding 
landmark points. A method of implementing MA that is not as frequently used is the 

10 visual matching of objects imbedded directly in the images being aligned. 

MA is usually carried out using a graphical user interface ("GUI"). In 
most cases, the GUI simultaneously displays the axial, sagittal, and coronal plane 
views of an area of interest. The GUI must provide efficient navigation of 3-D data, 
and precise localization of fiducial markers or other tools for alignment or measure- 

15 ment. Heretofore, commercial systems have used UNIX workstations or special-pur- 
pose computers to achieve the high computation and graphics throughput needed to 
interactively display large data volumes 

Automated image alignment ("ALA") methods involve obtaining a 
transformation through computation of the properties of the images to be registered. 

20 This may even take place through programmed computer actions without user inter- 
vention. Many of the more successful AIA methods are based on correlation and 
moment invariants matching. 

Correlation methods, more accurately, involve the cross-correlation or 
cross-covariance of image pixel intensities. These methods produce robust and accu- 

25 rate alignments of images. Methods of this type have been reported in Anuta, RE., 

"Spatial Registration on Multispectral and Multitemporal Digital Imagery Using Fast 
Fourier Transform Techniques " IEEE Transactions o n Geoscience Electronics, 8:353- 
368, 1970; Barnea, D.I., et al, "A Survey of Image Registration Techniques," IEEE 
Transactions on Computers , 21:179-186, 1972; and Pratt, W.K., "Correlation Tech- 

30 niques of Image registration," IEEE Transactions on Aerospace and Electronic Sys- 
tems . 10:353-358, 1974. 

According to most correlation methods, the location of the correlation 
peak will correspond directly to the translation needed to align images that already 
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have a correct rotational alignment. The rotational alignment may have been obtained 
by correlating the images after resampling them on a polar grid according to the 
method reported in Hibbard, L, et aL, Science236: 1641- 1646 , 1987. This rotational 
alignment method depends on the use of a fast Fourier transform. Correlation methods 

5 are accurate and robust to noise and differences in image content as long as the images 
are not too different. Accordingly, correlation methods are most often used for the 
alignment of images of the same kind. 

Moment invariants methods are computed methods for image registra- 
tion. A example of these methods is reported in Jain, A.K., "Fundamentals of Digital 

10 Picture Processing, Prentice-Hall . Englewood Cliffs, N.J., 1989. This moment invari- 
ants method involves the use of the principal moments computed from a two-dimen- 
sional C 4 2-D") inertia matrix of some prominent object of the images. The principal 
moments correspond to a unique set of principal vectors. Image alignment is per- 
formed by transforming one image's principal vectors onto the other image's principal 

15 vectors. This usually is a fast calculation. However, the moment invariants method 

depends on the ability to efficiently extract the object or set of objects that serve as the 
moment fiducials. This method has considerable problems if the images are complex 
and have no dominant single feature for the alignment of images of the same kind or 
modality. 

20 Besides the moment invariants method just discussed, there are other 

alignment methods based on invariant properties. These other methods may use geo- 
metric invariants, and minima/maxima of curvature and other extrema. At least one of 
these methods is reported in Lavellee, S., "Registration for Computer Integrated Sur- 
gery: Methodology, State of Art, in Taylor, R., et al, Computer Integrated Surgery, 

25 MIT Press, Cambridge, MA, 1996. The characterization of these methods are that they 
are fast but susceptible to noise and inter-subject differences. These systems, however, 
only use a small portion of all of the available information from each image set to com- 
pute the registration, which has some effect on the accuracy. 

In a practical sense, the automated alignment of CT and MRI images of 

30 the head was accomplished in the late 1980's and early 1990's. The methods that were 
used at that time gained some level of use but they had drawbacks. These methods 
were reported in Pelizzari, C.A. et ai, "Accurate Three-dimensional Registration of 
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CT, PET, and/or MR Images of the Brain, 7 " Journal of Computer Assisted Tomography , 
13:20-26, 1989. 

The method described in Pelizzari, et al suggests representing the sur- 
face of the brain in CT and MRI by stacks of contours of the cortical surface and mini- 

5 mizing the least-squares differences between neighboring points in the two surfaces to 
determine the alignment transform. This method was later refined by sub-sampling the 
surface points and then applying simple optimization to determine the transformation. 
This late refinement was reported in Van Herk, M. and Kooy, H.M., "Automatic Three- 
Dimensional Correlation of CT-CT, CTMRI, and CT-SPECT Using Chamfer Match- 

10 ing," Medical Phvsics . 21:1 163-1 178, 1994. It was found that the original and refined 
methods were useful only for the head, and both methods required that the cortical sur- 
face be manually contoured first. 

Another automatic method that has been suggested for the registration 
of images is based on the maximization of mutual information and this method was 

15 first reported on in 1995. This method has its genesis in information theory in which 
the relatedness of one random variable for another is based on the measure of the vari- 
ables' entropies, which is referred to as mutual information ("MI"). (See, Cover, T and 
Thomas, J, Elements of Information Theory . John Wiley and Sons, New York, 1991). 
Thus, for two images, the MI is small if the images are unrelated, or related but unreg- 

20 istered. If the images registration is improved, their MI increases and is maximized 

when the images are geometrically aligned. The MI is different from correlation in that 
systematic differences between images which would confound correlation actually 
strengthen the alignment by MI. MI is frequently used for two- and three-dimensional 
alignment of multimodal imagery in medical science as reported in Wells, W.M., et al, 

25 "Lecture Notes on Computer Science," Vol. 1496, Springer- Verlag, New York, 1998. 

The use of MI has also been the subject of a number of studies. For 
example, it has been used for aligning multimodal MRI and MRI with CT as reported 
in Viola, P. and Wells, W.M., "Alignment by Maximization of Mutual Information," 
Proc. of the Vth InTL Conf. on Computer Vision , Cambridge, MA, June 1995, pp. 16- 

30 23; Collignon, A., et al. "Automated Multimodality Medical Image Registration 

Using Information Theory." in Bizais, Y, et al, Proc. of the XVth Intl. Conf. on Com- 
puter Vision. Virtual reality, and Robotics in Medicine (CVRMed *95) , Vol 905, 
Springer- Verlag, Lecture Notes in Computer Science . Nice, France. April 1995, pp. 
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195-204; Meas, F, et aL "Multi-Modality Image Registration by Maximization of 
Mutual Information," IEEE Workshop on Mathematical M ethods in Biomedical Image 
Analysis . San Francisco, CA, June 21-22, 1996, pp. 14-22; and Studholme, C, etaL, 
"Automated 3-D Registration of MR and CT Images of the Head," Medical Image 

5 Analysis , 1: 163-175, 1996. Among other things, the issue with MI, however, is the 
ability to perform it in a time effective manner. 

In 1997, the Vanderbilt Retrospective Brain Registration study was con- 
ducted. This study was directed to computed alignment methods with the goal of 
assessing the error of each of the standard group of head data sets comprising CT+MRI 

10 or PET+MRI data sets in which "gold standard" alignments (alignments by which the 
others were judged) were measured using bone-implanted markers. In the study, evi- 
dence of the markers were removed from the images before being sent to the partici- 
pating laboratories. The techniques used by the testing laboratories included variations 
of the surface matching, correlation, and ML The errors in respective results varied 

15 greatly. There were at least fourteen methods used to align CT and MRI images. Two 
of the laboratories that used MI achieved the lowest total median error over the spec- 
trum of MRI pulse-sequence modalities. The third best results were achieved by a cor- 
relation method. In a comparison of alignments using PET and MRI, the first and 
second lowest total median errors were achieved by a single correlation method fol- 

20 lowed by two MI methods. In summary the MI and correlation methods produced the 
lowest errors and did not require brain contouring, scalp removal, or other interactive 
activities. Although these methods showed some success, it is believed that the time to 
solution could be very much improved without sacrificing accuracy. 

The present invention provides a system and method that includes an 

25 automated alignment method that is accurate but much faster that the prior art systems 
and methods; and the system and method of the present invention includes an interac- 
tive alignment method that achieves speeds and displayed detail equal to, or exceeding, 
the highest performance level of the prior art, without the need of specially configured 
or extremely powerful high performance workstations. 

30 

Summary Of The Invention 

The present invention is a system and method for autofusion and in par- 
ticular autofusion of three-dimensional ("3-D") image data volumes. According to the 
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system and method of the present invention, registration is effected by simultaneously 
displaying on a GUI at least, but preferably, two 3-D, image data volumes, and one of 
the 3-D image data volumes is held fixed while the other may be scaled, rotated, and 
translated to align homologous anatomic features. The system and method will simul- 

5 taneously display the axial, sagittal, and coronal plane views of a region of interest. 
Preferably, these plane views will be data volumes that are designed and configured for 
a system user to "drag" or "position" one volume to a new position in one plane view 
which will also simultaneously update the data volumes as displayed in the other two 
plane views with no apparent lag time. The system and method of the present invention 

10 also includes automated alignment computation based on mutual information ("MI") 
maximization. The system and method of the present invention using MI maximization 
aligns 3-D more efficiently and faster than prior image alignment methods. 

According to the system and method of the present invention, the dis- 
play and automated alignment referred to above have that advantage of achieving high 

15 speed updates of the data volumes on the display while at the same time displaying the 
data in full resolution. The system and method accomplish this following specific strat- 
egies. First, the composite transformations of two independent data volumes are repre- 
sented as a pair of homogeneous matrices whose product directly generates the point in 
one system homologous to a given point in the other system. Second, the volume data 

20 transformation is recast from matrix multiplication of real- valued vectors to the addi- 
tion of integer vectors that have components which approximate the real- valued vector 
components. Third, there is a determination of that portion of a row or column of data 
that will be visible for display or lie in the space of the fixed data, for automated align- 
ment, before performing the transformation and resampling. 

25 Autofusion according to the present invention is characterized as the 

alignment and combined use of multiple kinds of images of a region of interest This 
combined use of the multiple kinds of images permits the extraction of information 
about the region of interest that would be unavailable from any one of the images. As 
such, the registration or alignment process is one that spatially matches homologous 

30 objects of various images and a process for extracting synergistically derived informa- 
tion. 

Autofusion of the present invention is carried out with initial manual 
selection of images to be aligned, followed by either manual alignment using the GUI 
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or by automated alignment. The alignment methods may be used in either order, or 
repeatedly in any order, without restriction. More specifically, this involves the interac- 
tive matching of image objects using a graphical user interface ("GUI") and an auto- 
mated method to compute the estimated alignment of the two images without user 
interaction. These combined actions according to the present invention are comple- 
mentary and the alignment that is produced is more accurate and time effective than 
prior methods. 

The manual selection and initial superimposition of the two images is 
done with the GUI. The automated alignment uses a computed MI maximization 
method that automatically scales, and finds the translations and rotations that will pro- 
duce the best correspondence of the images. At least one embodiment of the present 
invention contemplates that the geometric transformation that is applied to the movable 
data volume is for linear transformations involving six degrees of freedom: three trans- 
lations and three rotations. 

The system and method of the present invention will described in 
greater detail in the remainder of the specification referring to the drawings. 

Brief Description Of The Drawings 

Figure 1 is a perspective view of the GUI employing the present inven- 
tion that shows the unaligned CT and MRI images of the same patient in the trans- 
verse, coronal and sagittal planes. 

Figure 2 is a perspective view of the GUI employing the present inven- 
tion that shows the CT and MRI images in the transverse, coronal and sagittal planes 
from Figure 1 aligned. 

Figure 3 is a representative drawing of a CT stack containing an image 

data volume. 

Figure 4 is a representative drawing of a MRI stack containing an image 

data volume. 

Figure 5 is a representative drawing of the data volume coordinate sys- 
tem for use in the present invention that shows the transverse (axial), coronal and sagit- 
tal planes. 

Figure 6 is an example of a primary and secondary data volumes having 
different scales. 
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Figure 7 is an example of primary and secondary data volumes having 
scaling and translation differences. 

Figure 8 is an example of primary and secondary data volumes having 
scaling, translation and rotation differences. 

Figure 9 is an example of data interpolation for the secondary data vol- 
ume on the primary data volume. 

Figure 10 is an example of the transformation of a secondary data vol- 
ume onto a primary data volume by iterated, integer- vector addition. 

Detailed Description Of The Drawings 

The present invention is a system and method for autofusion of three- 
dimensional ("3-D") image data volumes. According to the system and method of the 
present invention, autofusion is effected by simultaneously displaying on a GUI at 
least, but preferably, two 3-D, image data volumes, and one of the 3-D image data vol- 
umes is held fixed while the other may be scaled, rotated, and translated to align 
homologous anatomic features. The system and method will simultaneously display 
the axial, sagittal, and coronal plane views of a region of interest. Preferably, these 
plane views will display the voxel intensities on 2-D planes within the data volumes, 
whose locations within the data volume are specified by the locations of straight-line 
cursors in the GUI axial, coronal, and sagittal windows, such as "dragging" a cursor in 
any window specifies a new location for one of the plane views, and that plane view is 
immediately updated with no lag apparent to the user. In a second method of operation, 
one of the two data volumes may itself be "dragged" with a cursor to effect a rotation 
or translation with respect to the fixed data volume. This will cause all three views, 
axial, coronal, and sagittal, to be re-displayed with no lag apparent to the user. In a 
third method of operation, the movable data volume may "jump" and "twist" in real- 
time response to the latest, "best," alignment obtained during the maximization of 
mutual information determination. Autofusion of the present invention also includes 
automated alignment computation based on mutual information ("MI") maximization. 
The system and method of the present invention using MI maximization of aligns 3-D 
more efficiently and faster that prior alignment methods. 

Autofusion of the present invention is carried out with an initial manual 
selection of images to be aligned, followed by either manual alignment using the GUI 
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or by automated alignment. The alignment methods may be used in either order, or 
repeatedly in any order, without restriction. That is, first, there is interactive matching 
of image objects using a graphical user interface ("GUI") and then there is an auto- 
mated method to compute the estimated alignment of the two images without user 

5 interaction. This combination produces more accurate alignments than prior methods. 

Autofusion, according to the present invention, involves the registration 
of different image volumes by interactive matching of image objects using a GUI and 
automated computation of alignment. The use of the GUI provides the system user 
with a three views of the 3-D data volumes at issue, and permits the system user to 

10 translate and rotate the image data of one modality with respect to the other. For exam- 
ple, the desire may be to autofuse CT and MRI studies. For the purposes here each 
study is in the form of a data volume. The CT image data will be in a fixed position of 
the GUI while the MRI image data may be rotated and translated with respect to the 
CT image data. In this case, the CT image data is referred to as the primary data set and 

15 the MRI image data is referred to as the secondary data set. The primary and secondary 
data sets will have window and level controls on the GUI. There also will be a slider 
control at the GUI for controlling the intensity of the primary and secondary data sets 
in the windows. 

The display (through the GUI) and automated alignment achieve both 
20 high speed updates of the GUI while displaying the data at full resolution. Autofusion 
of the present invention depends at least in part on (/) representing the composite trans- 
formations of the two independent data volumes as a pair of homogeneous matrices 
whose product, with that of the transformation, directly generates the point in second- 
ary system homologous to a given point in the primary system; (ii) the volume data 
25 transformation is recast from matrix multiplication of real-valued vectors to the addi- 
tion of integer vectors whose components approximate the real vector components; 
and (iii) determining the portion of a row or column of data which will be visible on 
the display or will lie in the space of the fixed data (for automated alignment) before 
performing the transformation and resampling so that needless transformation are not 
30 carried out. 

In the practical application of the present invention, autofusion will 
align two 3-D image data volumes for use in RTP. Typically, this will consist of align- 
ing one X-ray CT study set and one MRI study set. The alignment of the images of 
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these two modalities is to augment the CT anatomic information with greater soft tis- 
sue detail that is seen in MRI imagery: the contours of soft tissue organs and tumors 
after better shown in MRI than in CT. If the two modalities are accurately superim- 
posed, the MRI images, where needed, can substitute for what is not shown or not 

5 shown well in the CT images. Autofusion of the present invention may also employ 
specific tissue functional information from nuclear medicine imagery (SPECT, PET) 
or functional MRI imaging (fiVIRI) which may be helpful to obtain more accurate and 
effective treatment. 

Referring to Figure 1, generally at 100, a representative GUI is shown 

10 for use in the present invention. Then present invention is preferably for aligning a pri- 
mary (CT) and secondary (MRI) image data to aid in contouring tissue, although many 
other uses are contemplated by the inventor. 

In Figure 1, the transverse or axial plane view of the primary and sec- 
ondary data sets is shown at window 102. The primary data set is shown at 104 and the 

15 secondary is shown at 106. The coronal plane view of the primary and secondary data 
sets is shown at window 108. The primary data set is shown at 1 10 and the secondary 
is shown at 112. The sagittal plane view of the primary and secondary data sets is 
shown at window 114. The primary data set is shown at 116 and the secondary is 
shown at 1 18. In each of the windows, the primary and secondary data sets have con- 

20 trasting colors or one set could be colored and the other black and white. However, all 
that is necessary is that the two sets be colored so that they may be distinguishable. 
This is necessary so the user can tell how the secondary is being rotated and translated 
with respect to primary. As is readily seen by reviewing windows 102, 108, and 114 
and the contrasting colors of the two data sets, the primary and secondary data sets are 

25 unaligned. The blank area 132 will remain this way until MI maximization is per- 
formed at which time the MI levels will be displayed as will be discussed subse- 
quently. 

Again referring to Figure 1, the primary window control is shown at 
122 and primary level control is shown at 124. Primary window control 122 is used to 
30 select the window from the primary data set that will be displayed in windows 102, 
108, and 1 14. As such, the number 1301 indicates that the primary data in an interval 
1301 intensity units wide, centered on the level value of 164, is displayed. Primary 
level control 164 is used to set the intensity level for the primary data set displayed in 
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windows 102, 108, and 1 14. Thus, the number 164 indicates that the primary data in an 
interval 1301 intensity units wide, centered on the level value of 164, is displayed. 

In a similar fashion, the secondary window control is shown at 126 and 
secondary level control is shown at 128. Secondary window control 126 is used to 

5 select the window from the secondary data set that will be displayed in windows 102, 
108, and 114. As such, the number 100 indicates that the secondary data in an interval 
100 intensity units wide, centered on the level value of 50, is displayed. Secondary 
level control 128 is used to set the intensity level for the secondary data set displayed 
in windows 102, 108, and 114. Thus, the number 50 indicates that the secondary data 

10 in an interval 100 intensity units wide, centered on the level value of 50, is displayed. 

At the bottom of GUI 100 is a slider with the labels "Primary" and 
"Secondary" at respective ends. Movement of the control will increase and decrease 
the intensity of the secondary with respect to the primary. Accordingly, if the user 
wants to emphasize the primary data set more than the secondary data set, he or she 

15 will move the control closer to the "Primary" label; and if the user wants to emphasize 
the secondary data set more than the primary data set, he or she will move the control 
closer to the "Secondary" label. The use of this control is important in selecting and 
aligning the primary and secondary data sets on the GUI. 

Referring to Figure 1, windows 102, 108, and 114 each have a circle 

20 and two line-cursors. More specifically, window 102 has circle 134, vertical line-cursor 
136, and horizontal line-cursor 138; window 108 has circle 140, vertical line-cursor 
142, and horizontal line-cursor 144; and window 114 has circle 146, vertical line-cur- 
sor 148, and horizontal line-cursor 150. The two line-cursors in each window depict 
the locations of the other two plane views in data volume. Referring to window 102, 

25 horizontal line-cursor 138 shows the location of the coronal section shown in window 
108 and vertical line-cursor 136 shows the location of the sagittal section shown in 
window 114. Similarly, referring to window 108, horizontal line-cursor 144 shows the 
location of the transverse or axial section shown in window 102 and vertical line-cur- 
sor 142 shows the location of the sagittal section shown in window 114. Finally, refer- 

30 ring to window 1 14, horizontal line-cursor 150 shows the location of the transverse or 
axial section shown in window 102 and vertical line-cursor 148 shows the location of 
the coronal section shown in window 108. 
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Movement of vertical line-cursor 136 in window 102 will cause the sag- 
ittal plane view in window 1 14 to pan and movement of horizontal line-cursor 138 in 
window 102 will cause the coronal plane view in window 108 to pan. Likewise, move- 
ment of vertical line-cursor 142 in window 108 will cause the sagittal plane view in 
5 window 1 14 to pan and movement of horizontal line-cursor 144 in window 108 will 
cause the transverse or axial plane view in window 102 to pan. Lastly, movement of 
vertical line-cursor 148 in window 1 14 will cause the coronal plane view in window 
108 to pan and movement of horizontal line-cursor 150 in window 102 will cause the 
transverse or axial plane view in window 102 to pan. The movement of these line-cur- 

10 sors permit the user to navigate the data volumes that are shown, through the simulta- 
neous updating of the affected views with each movement of the line cursors. 

Circles 134, 140, and 146 in windows 102, 108, and 114, respectively, 
indicate the areas where the user may engage a particular window view to manually 
rotate or translate the secondary image. Thus, the GUI cursor, which is to be distin- 

15 guished from the vertical and horizontal line-cursors, is positioned within and without 
the a circle to effect rotating and translating the MRI data set (the secondary data set). 
If the GUI cursor is placed within the circle and engaged, it may be used to translate 
the secondary image in two directions in the plane of the window. This translation will 
be evidenced in the other two plane views simultaneously. If the GUI cursor is placed 

20 outside of the circle and engaged, it may be used to rotate the secondary image about 
the center of the plane of the window. The rotation will be evidenced in the other two 
plane views simultaneously. 

When two data volumes for registration are selected, the first action is 
the manual manipulation of the unaligned views. This will involve the selection of the 

25 proper primary and secondary window views, and the desired intensity levels for these 
two window views. The relative intensity between the primary and secondary images 
may be controlled by slider control 130. Once the proper views and levels are selected, 
the line-cursors of the respective windows are used to obtain suitably informative 
views of the primary and secondary data sets in each of the windows. Following this, 

30 the GUI cursor is positioned within and without the circles of each window to translate 
and rotate the secondary images for alignment of the secondary (movable) image with 
the primary (fixed) image. 
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Automated alignment, using MI maximization, may be performed 
before, after, or instead of, manual alignment. Automated alignment is initiated by 
activation of automated alignment control shown at 152 in Figure 1. The activation of 
the automated alignment control will cause the improved alignment of the secondary 

5 image on the primary image. 

Figure 2, shown generally at 200, shows windows 102, 108, and 114 
with the primary and secondary image aligned or registered. In Figure 2, the elements 
that are the same as those in Figure 1 have the same reference numbers. The alignment 
of the primary and secondary images in Figure 2 is the result of automated alignment 

10 techniques according to the present invention. It is to be noted that the alignment that 
takes place is with respect to all three of the plane views, transverse (or axial), coronal, 
and sagittal, simultaneously. 

The progress and accuracy of the automated alignment method is 
shown in window 156 which contains graph 158 that shows a plot of the value of the 

15 mutual information versus iteration number. The final MI maximization value is dis- 
played at 160 of window 156. This later value is a very accurate registration of the pri- 
mary and secondary images so that useful information from each image may be use, 
for example, in RTR As automated alignment is being performed, the image displays 
are updated to reflect the incremental improvements in the alignment. The method by 

20 which automated alignment of the primary and secondary image is carried out will 
now be describe 

Both the display and automated alignment of the present invention 
require rapid determination of CT or MRI intensities at arbitrary locations in the 3-D 
primary and secondary data volumes. According to the present invention, translation 

25 and rotation of a data volume in a single window will result in the computation of the 
position in the corresponding plane in the data volume. Correspondingly, the intensi- 
ties are determined for that plane position. Since digital image data occurs only at a 
finite number of rows, columns, and slices, the viewing lines located between the rows, 
columns, or slices must estimate the intensities. Likewise, the automated alignment 

30 performs a voxel-by-voxel comparison of the overlapping parts of the primary and sec- 
ondary data of the data volumes. This requires comparing primary voxels at integer 
row, column, and slice coordinates with transformed secondary voxels at non-integer 
locations in the primary coordinate system. Accordingly, the present invention has the 
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ability to determine transformed locations and intensities at new locations very rapidly, 
preferably at a rate of several million or greater per second, to have the GUI views 
updated without visible lagging. This is accomplished by casting the geometric trans- 
formations in an efficient and compact representation and substituting for the time 

5 costly multiplication of real (decimal) numbers with serial addition of integers. More- 
over, according to the present invention, integer rescaling of the transformations of the 
secondary data volume is done by bit shifting which facilitates faster results. 

To improve the time efficiency of the method of the present invention, 
there is a determination whether the volume data to be transformed remains in the 

10 domain of the display or the overlap volume. The present invention only performs 

transformations on those voxels which contribute usefully to these functions. Further, 
to increase the speed of the results, scan line visibility checking may be accomplished 
before transformation takes place since the geometries of these data and the display 
coordinate system are known so transformation is not necessary. 

15 As background before discussing the autofusion of the present inven- 

tion, the CT and MRI data sets that preferably are used, for example, in RTP are sets of 
samples of a patient's anatomy at regular intervals on a 3-D rectangular grid. CT 
images represent sections of the anatomy reconstructed from absorption measurements 
obtained as projections through the patient. The reconstruction plane pictures form the 

20 transverse or axial plane view of the patient. Normally, CT studies have a spatial reso- 
lution of 0.7 to about 1 mm/voxel (volume element) in the plane of the reconstructed 
section images. Typically, CT images have dimensions of 512 x 512 voxels in the sec- 
tion planes and may have 40 to greater than 100 sections for a study. 

MRI images are reconstructed from volumetric magnetization measure- 

25 ments. These images are reconstructed along planes normal to the axis of the patient, 
so, normally, the images set is formed in the transverse or axial orientation and, there- 
fore, provide comparisons with CT imagery of the same patient. MRI spatial resolu- 
tions are approximately 1.5 to 3 mm/voxel in the plane of the reconstructed images. 
These MRI images normally have dimensions of 256 x 256 voxels in the section 

30 planes. Typically, MRI studies have 30 to greater than 100 sections. 

The most common presentation of both CT and MRI images is a stack 
of images showing sections of the patient's anatomy in planes perpendicular to the 
scanner's axis, and the patient's inferior-superior axis. A representative CT stack 
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which includes a data volume is shown in Figure 3 generally at 270 and a representa- 
tive MRI stack which includes a data volume is shown in Figure 4 generally at 272. 
The images along the patient's inferior-superior axis are referred to as transaxial (or 
axial or transverse). Images in planes perpendicular to the anterior-posterior direction 
5 are coronal and sagittal images, and are in planes perpendicular to the medial-lateral 
direction. For description purposes only, the following example of CT and MRI images 
data sets is provided for a patient that will be using RTP: superior - toward the head; 
inferior - toward the feet; anterior — toward the top or front of the body or toward the 
face; posterior - toward the back of the body or head; medial - toward the center of 
10 the body or head; and lateral - toward the exterior side of the body or head. 

Assuming a supine position of a patient when a CT or MRI scan is per- 
formed to produce the data stacks shown in Figures 3 and 4, the data volume coordi- 
nate system showing the transverse or axial, sagittal, and coronal planes, along with 
the x, v, and z axes is shown in Figure 5 generally at 274. In Figure 5, the axial plane is 
15 shown at 275, and the sagittal plane at 276, the coronal plane at 277. The x, y, and z 
axes are shown generally at 278. 

Image data consists of scalar intensities at regular intervals on a 3-D 
right rectangular grid with orthogonal coordinate axes (x,y,z). The z axis is parallel to 
the inferior-superior axis or perpendicular to the transverse or axial image sections. 
20 The x axis is parallel to the anterior-posterior axis and perpendicular to the coronal 

image sections. Finally, the y axis is parallel to the medial-lateral axis and perpendicu- 
lar to the sagittal image sections. The x and y axes lie in the transverse or axial image 
plane with the origin at the upper left of the image. The x axis is parallel to the medial- 
lateral axis and perpendicular to the sagittal image sections. The y-axis is parallel to 
25 the anterior-posterior axis, and perpendicular to the coronal image sections. 

Preferably, the image data that is being processed according to the 
present invention is digital data. As such, the CT and MRI data for the data volumes 
will be an array of numbers with each number representing a measurement or sample 
of a tissue property at a point in space. These data points are taken to exist at the cen- 
30 ters of voxels which are rectangular volume with a length, width, and height equal to 
the characteristic spacing between data. The geometry of the data is consistent with a 
Cartesian coordinate system that has an origin and axes at right angles to each other. 
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The automated alignment portion of the autofusion method of the 
present invention is for automatically performing secondary image transformation, 
which in this case is meant to mean three rotations and three translations of the second- 
ary image with respect to the primary image that will maximally superimpose homolo- 
gous anatomical points. This optimal transformation, Z 7 *, corresponds to the maximum 
value of the mutual information represented by I(p,F(s)) which relates that primary 
image data to the transformed secondary image data according to the Expression (1): 

F* = maxl(p, F(s)) (D 
F 

where, 

/ = is the real value of the ML 

p = is an array of the primary intensities. 

F(s) = is the array of the secondary intensities transformed by F 

In most cases, the real-valued rotations and translations relocate the 
integer-indexed secondary voxels to non-integer locations. Accordingly, the secondary 
image data must be interpolated to obtain secondary voxel intensities at the primary 
voxel locations. These interpolated F(s) intensities are to form the marginal and joint 
probability distributions that are needed to determine the ML 

The determination of the MI maximization is dependent on an evalua- 
tion of the transformations, interpolation, and probability distributions underlying 
Expression (1). These transformations, interpolation, and probability distributions will 
now be discussed. 

First the coordinate transformations must be considered. A transformed 
secondary data voxel may be represented by s t . This value is transformed from the ini- 
tial state of a secondary data voxel which is represented by s=(s x , s } , s z ). The transfor- 
mation is accomplished by a combination of scaling, translation, and rotation 
operations. This transformation is represented by Expression (2): 
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s f = R(v-s-t) (2) 



is (v X9 v z ), the vector of the primary to secondary axis scale 

is (<p x , (py, cp z ), the matrix of rotations about the three axes. 
is (t x , ty t z ), the translation vector. 

is (s x , s y s z ), the initial secondary data voxel coordinates before 
scaling, translation, or rotation. 

The representative relationship of the primary and secondary data of the 
15 data volumes prior to scaling or translation is shown in Figure 6, generally at 300. In 
Figure 6, the primary and secondary data volumes are shown with the same origin at 
302 so that the differences may be readily seen. The primary data is shown at 304 and 
the secondary data at 306. It also is readily seen that the x and y spacing for the primary 
and secondary at 308,310 and 312,314, respectively is different. The differences 
20 between the primary data volume and the secondary data volume in Figure 6 is due to 
scaling differences. As such, before any further alignment can be made, it also is nec- 
essary for the secondary data volume to be scaled to primary data volume. 

The vector v x , v y v z is known from scanner calibrations of the image 
data and it is considered fixed. Therefore, it is only necessary to vary the translations 
25 and rotations to obtain an optimal alignment or registration once scaling has taken 
place. 

The necessary coordinate transformations may be performed using 
matrix multiplication 

as set forth below. In this case, these transformations may be presented by matrix oper- 
30 ations in homogeneous coordinates in which vector translation, rotation, scaling, and 
skewing operations are combined in one notation. In the three dimensions of the data 
volume, vectors are represented as 4 x 1 arrays according to Expression (3): 



where, 



10 



v 

ratios. 

R 

t 
s 
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s = (s x , s y , s z ,l) = 



y 

s z 
1 



(3) 



10 



15 



where, 

s = the initial vector for the secondary data voxel. 

T = transpose of the vector. 

Considering Figure 6, in order to compare the intensity at a point a of 
the primary image data with the same point in the secondary image data requires inter- 
polating the secondary image data values. The secondary distances D s are scaled to the 
primary distances D p , and this may represented by Expression (4): 



D P D s 



(4) 



The corresponding scaling components are set forth in Expression (5): 



20 




The corresponding transformation is shown in Expression 6 which will be discussed 
subsequently. 

The linear and affine transformations, such translations, rotations, scal- 
ing, and skewing, are represented by 4 x 4 matrices. More particularly with regard to 
scaling, this is achieved by the multiplication of the vector of scale factors v in a 4 x 4 
matrix (with the components of the vector v on the diagonal) with the 4 x 1 matrix 
shown in Expression (3). This is shown in Expression (6): 



30 
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V x 
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0 








V x S .v 




vs = 
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V v 
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0 




s y 




Vy 






0 


0 


v z 


0 




s z 






5 




0 


0 


0 


1 




1 




1 _ 



(6) 



The determination of s t must made repeatedly for the plurality of sec- 
ondary data voxels. As such, for efficiency, it is convenient to pre-multiply s times v as 
shown in Expression (6). The determination of s' according to Expression (6) serves 
10 as a resampling equivalent for the secondary image data along all three axes so that the 
same grid spacing for the secondary image data is achieved as is present in the primary 
image data. 

Referring to Figure 7, generally at 320, shows a situation in which the 
secondary data volume differs from the primary data volume by scale factors and a 
15 * 

translation T. The translation difference is the sum of the component translations 
along x and y, and represented at 338 in Figure 7. 

Taking this into account, the determination of s t may be based on one 
translation and one rotation of the secondary image data about each of the three axes as 
20 is seen by Expression (7): 

s t = R(s'-t) (7) 

25 where, 

R = is (<p x , (fy cp z ), the matrix of rotations about the three axes. 

s' = the product of v and s. 

t = is (t x , t v t z ), the translation vector. 

The translation of s' by vector t=(t x , r r f.) is represented by the multi- 

30 

plication of this vector by the 4 x 4 matrix T with the translation components in the 
fourth column of the matrix, as set forth in Expression (8): 
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1 0 0 t x 




r 

S X 




S'x + 't 


0 1 0 t 

V 




/ 

S v 




S'y + fy 


0 0 1 t z 




r 

s , 




s' z + t. 


0 0 0 1 




1 




1 



(8) 



Next the rotations of the secondary image data must be considered. 
Referring to Figure 8, generally at 350, the combined offsets due to scaling, translation 
-» 

by T at 356, and rotation by angle 0 at 358 is shown. In Figure 8, the primary volume 
data is shown at 352 and the secondary volume data is shown at 354. The rotations of 
the secondary image data by the angles (p v , cp v , <p z about the three coordinate axes is 
according to Expressions (9), (10), and (11): 



15 



10 0 0 
0 cos<j) x -sine)),. 0 
0 sin^ cos§ x 0 
0 0 0 1 



(9) 



20 



cos(f> v 0 sin<)> v 0 

0 10 0 

-shi(|> v . 0 cos<]> v 0 

0 0 0 1 



(10) 



25 



30 



cos<j), -sin(|>, 0 0 
sin(j), coscj)^ 0 0 
0 0 10 
0 0 0 1 



(ID 



The rotation of secondary image data, s' , about the z-axis, which will 
be in thexv plane, is represented by Expression (12): 
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/*(4>,)s = 



eos(f>, -sinc|>, 0 0 

sin0, cos(|>, 0 0 

0 0 10 
0 0 0 1 



s r cos({>,-s v sin<|), 
5 t sin(() z + .s r cos(t>, 

s z 
1 



(12) 



10 



This same type of matrix multiplication is used for the other two axes, R(ty x )s and 

The combinations of the coordinate transformations may be expressed 
as the products of the individual transformation matrices. The combined transforma- 
tion equivalent to the translation followed by rotation of the secondary image data 
about the z axis is set forth in Expression (13): 



15 



s tr = R($ z )Ts = 



0 
0 



0 
0 
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1 0 0 t x 




s x 
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0 10f v 




s y 
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OOlf, 




s z 




0 0 0 1. 




1 



20 



cos <{), -sin<J), 0 0 

sin<}> T cosc^ 0 0 

0 0 10 
0 0 0 1 



s x~*x 



Sy-t y 
s z - t z 
1 



(13) 



25 



30 



(s x - t x )cos$ z -(s y - f y )sin<|> z 
(s x - t x ) sin<|) z + (s y - t y )costy z 

1 



Noting the determination of s tr above, the computations necessary for 
homogeneous coordinates may be inefficient unless specially designed hardware sys- 
tems are used since there are a large number of multiplications by zero and one. As 
such the present invention utilizes an approach that will produce accurate results with- 
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out the need for the level of complex computations that have been a problem in the past 
to perform in a time-efficient manner. This approach will now be described. 

The rotations of the secondary image data are all basically 2-D. Thus, 
they may be restated as 2 x 2 matrices containing only the sin and cos terms of Expres- 
5 sions (9), ( 10), and (11) multiplied by the appropriate combinations of x, y, z taken two 
at a time. These rotations also must be performed with reference to an origin or rota- 
tion center. Preferably, the rotation center is not the same point as the primary and sec- 
ondary data volume origins. 

As stated, the origins for the primary and secondary data volumes are 

10 at one corner of the rectangular volumes. The rotations of the secondary image data 
about its origin will introduce large and unpredictable effects in the attempt to find 
optimal alignment of the common features of the primary and secondary image data. 
As such, according to the present invention, the rotation center is set at the center of the 
secondary data volume so that the rotations will leave the secondary volume data 

15 largely superimposed on the primary volume data. To carry out this action, there are 
two translations that are of concern. The first is to translate the secondary data volume 
center to the coordinate system origin, which preferably will be defined with respect to 
the primary data volume. The second is to translate the secondary data volume back to 
its original position. The first translation is represented by Expressions (14) and (15): 

20 

s t — C l RCTs', C = 

25 

is the transformed secondary data voxel, 
is the matrix of the translation of the secondary volume data 
with its center at ( c x , c r c z ) to the coordinate system origin. 

is the inverse translation of the C matrix. 
is (cp v , <p v ., (p z ), the matrix of rotations about the three axes, 
the product of v and s. 
is the transpose of the vector. 



1 0 0 
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1 0 0 




0 1 0 


c y 


, c 1 = 


0 1 0 


~ c y 


0 0 1 


c. 




0 0 1 
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poo 
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0 0 0 


1 _ 



(14) 



30 



where, 

C 



R 



s 
T 
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Now expanding the rotation term, R, to embrace the rotations in the three axes, the 
total transformation is represented by Expression (15): 



s t = C R z R x R x CTs' 



(15) 



10 



As previously stated, the rotations about the jc, y, and z axes involve changes 
only to the two coordinates in the plane to that axis. Therefore, with regard to each 
axis, the information being considered is 2-D rather than 3-D. As a result, this will 
reduce the 4 x 4 matrix multiplications, considering Expressions (9), (10), (11), and 
(12), to a series of 2 x 2 matrix multiplications, which is shown in Expressions (16), 
(17), and (18): 



15 



cos$ x -sin()) x 
sin^ cos 4^ 



, u = s v , v = s 



(16) 



20 



R y s' 



cos<j) v sin<}> y 
-sin(() v cos<t> v 



, u = s x , v = s z 



(17) 



25 



R z s' 



cos<(), -sine)), 
sin^ cos<() z 



, u = s x , v - s v 



(18) 



where. 



30 



R y 

It 

V 



is the rotation about the jc axis in the v and z plane, 
is the rotation about the y axis in the x and z plane, 
is the rotation about the z axis in the x and y plane, 
is the first component, 
is the second component. 
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After the translation of the secondary data volume center has been 
made to the coordinate system origin according to the Expressions (16) - (18), then 
there will be the translation of the secondary data volume back to its original position. 
These will provide an error value which now may be used for the remaining automated 
5 alignment steps. 

As discussed, in carrying out the automated alignment portion of the 
autofusion method of the present invention, there most likely will be non-integer align- 
ment requiring interpolation to the optimal alignment position of the secondary data 
volume to the primary data volume. In principle, if the there are two real numbers, for 
10 example, a and b, where a<b, and a third number x s positioned between a and b, such 
that a<x<b . Given this, the value of x based on linear interpolation may be deter- 
mined by Expression (19): 



20 



Thus, linear interpolation in 2-D may be used to resample and reconstruct the second- 
ary volume data in the planes of the primary volume data. In order to effect alignment 
in the 3-D volume data, the 2-D information may be coupled with nearest neighbor 
voxel techniques to obtain the desired interpolation without requiring excessive com- 
putation time, although other interpolation methods may be used. 

Referring to Figure 9, generally at 370, there is a representation of a sit- 
uation in 2-D after transformation of the secondary volume data to the primary coordi- 
nate system. The value of the secondary data at point a, S(a\ may be determined by 
interpolation methods such as nearest neighbor, bilinear, B-spline interpolation, or 
higher polynomial methods. In the Figure, the primary volume data is shown at 372 
and the secondary volume data at 374. 

In general, the nearest neighbor method requires that the secondary 
point of interest may be assigned the value of the nearest secondary data point. This 
may be done by rounding the real valued components jc, v, and z to the corresponding 
nearest integer. Preferably, the present invention uses this interpolation method. 

Before discussing MI, an example of the method used to determine the 
transformations and interpolation, and perform the scan line analysis will be provided. 
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First the two coordinate systems must be related. The alignment of the two 3-D data 
volumes with different x,y,z scale factors implies the determination of the translations 
t x , ty, t z and the rotations ty x , fy, § z which make the corresponding anatomies coincide. 
Since the complete description of a data volume transformed relative to a fundamental 
5 Cartesian coordinate system may be represented in a matrix of its basis vectors, the 
location of a voxel in one system (the secondary system) "transformed to another sys- 
tem (the primary system) may be determined using the systems' basis vectors. There- 

fore, considering the CT image volume (primary) with basis matrix P, origin offset op , 
„ and basis vectors p*, p v , Pz , the basis matrix P is according to Expression (20): 



15 



P = 



P\xPlx P$x °Px 

Ply Ply Ply °Py 

Plz Plz Piz °PZ 
0 0 0.1 



(20) 



The accumulated scaling and rotations are contained in the upper left 3 
x 3 submatrix that is shown in Expression (20). If the origin and axes of the secondary 

volume data coincide with the fundamental Cartesian coordinate system i, j, k of the 
primary volume data, then basis matrix is according to Expression (21): 



25 



P = 



Plx 



0 0 0 
0 p 2y 0 0 
0 0 p 3z 0 
0 0 0 1 



(21) 



-> -» ~> 



In Expression (21), p lx ,p 2y > P3 Z are the lengths of thep*, p>, p z vectors projected on 
the i, j, k axes. The vector p for the voxel at [rows] [columns] [slices] = [/][m][n] in 



30 



this system has the form shown in Expression (22): 
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p = Pp 



Imn 



= p 



= lp lx i + mp 2 j + np 3z k 



(22) 



In Expression (22), the coordinates are shown as integral multiples of the length of the 
basis vectors. 

In the more general system expression at Expression (20), the vector 
pimn may be shown as Expression (23): 



p/mn = P 



Plx Plx Plx °Px 

Ply Ply Ply °Py 

Plz Plz P3z °Pz 
0 0 0 1 



lp lx + mp 2x + np 3x + o Px 
Ip ly + mp 2y + np 3y + o Py 
lp lz + mp 2z + np Zz + o Pz 
1 



(23) 



In Expression 23, the components of p/^ on the i, j, k axes are shown in the last term. 

-» ~> > > -> 

Each of the basis vectors p,r, p>, Pz have components on all the i, j, k axes thereby 

requiring the use of a more complicated expression than Expression (22). Expression 
(23) provides the physical location of voxel [/][m][n] given the integers /,m,«. If matrix 
P has an inverse P" 1 , then the integer location of a voxel may be determined from its 

physical location vector p per Expression (24): 



25 



pimn - P p 



(24) 



Noting the foregoing in this example, a secondary image volume may 
be represented by a homogeneous basis matrix S as shown in Expression (25): 
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S = 



; \x S 2x S 3x °Sx 



S lz S 2z S 3>z °S: 

0 0 0 1 



(25) 



10 



* * * > 

Expression (25) reflects both the origin offset os and basis vectors s x , s>, s z . Like the 

primary system, the equivalence of the physical locations in the secondary system and 
the integer indexed voxels of the secondary volume data are established by the basis 
matrix according to Expressions (26) and (27): 



S = SSbnn 



(26) 



15 



* ~i* 

Slmn = S S 



(27) 
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M represents a general homogeneous affine transformation matrix. This 
is shown in Expression (28): 







m \x 


m 2x 


m 3x 


t x 
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M = 


m ly 
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m 3y 








m \z 


m 2 . 
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1 



(28) 



where, 



30 



,H \ x t}t ~>x fll 3x 
"My '"2v W 3y 



=is the upper left 3x3 sub-matrix of M containing scaling and 



rotation information. 



f v , ty, t z = the translation vector 
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5 



The M matrix of Expression (28) provides that any number of transformation may be 
combined by multiplying matrices with what is shown as m Y , m y m z . 

The transforming of the secondary volume data is equivalent to the 
matrix product shown in Expression (29): 



S' = MS (29) 



10 The location s' of the transformed voxel vector s/,„» has the fundamental location that 
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25 



30 



is set forth in Expression (30): 



s' = MSs/mn = S's/,„« (30) 



In order to display of the intensities of the transformed secondary vol- 
ume data in the primary volume data coordinate system or to compare secondary voxel 
intensities with primary voxel intensities, there must be a determination of the location 

> > ■> 

s' in the primary coordinate system. To do this, it is necessary to substitute s' for p in 

Expression (24) and the resulting expression is Expression (31): 



p' = P 1 ^' = P _1 MSs/„m (31) 



Expression (31) will provide the location p' in the primary coordinate system. Thus, 
as the steps of the method of the present invention are performed through the second- 
ary volume data lattice, a corresponding lattice of transformed points is determined in 
the primary coordinate system but at non-integer locations relative to the primary vol- 
ume data lattice. 

The full product of the basis matrices and the transforming matrix in 
Expression (31) is the homogeneous matrix in Expression (32): 
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P l MS = X = 



X l\ X 2\ X 3l X 4\ 
X \2 X 22 X 32 X 42 
X \3 X 23 X 33 X 43 



14 x 24 X 34 X 44 



(32) 



10 



15 



Considering Expression (28) above, the upper left 3x3 sub-matrix of 
this Expression contains the accumulated product of the rotations about the three axes 
and the first three elements of the fourth column contain the components of the transla- 
tions relating the secondary to the primary system origin. In addition, the first the ele- 
ments of the first three columns are the basis vectors for the transformed coordinate 
> > 

system relative to the i, j, k axes ot the primary coordinate system. The origin offset of 

-> * ~> * 

the transformed secondary with respect to the primary is o = o.r + o v + o z and shown 
in Figure 10, generally at 400. In Figure 10, the primary volume data is at 402 and the 
secondary volume data is at 404. Since Figure 10 is only a 2-D representation, in the 
x,y plane, it follows that only the x offset at 408 and y offset at 410 are shown. The ori- 
gin offset is obtained from X from Expression (32) according to Expression (33): 



20 



Ox = X 41 i, Oy — -*42«j> °Z ~ x 43^ 



(33) 



* » * 

The set of the three basis vectors s*, s v , s z is found in Expressions (34), (35), and (36): 



25 



(34) 



30 



Sv — *2l' + x 22$ x 23^ 



(35) 
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s z = x 31 i + * 32 j + x 33 k (36) 

5 

Expressions (34)-(36) define the integral stepping away from o along 

the transformed secondary axes Sx, s> as shown in Figure 10. Stepping along the rows, 
columns, and slices away from the origin, any successive secondary voxel location 
may be determined by adding the appropriate vector components. Accordingly, the 
10 four data points a at 412, b at 414, c at 416, and d at 418 will have the locations indi- 
cated below: 

a: lSx + Osv 

b: lsx+ ls y 

15 > > 

c: lsx + 2s v 

> > 
d: lsx + 3sy 

Thus, the transformed secondary locations are obtained by iterated addition of real- 
time vectors. Further, greater processing time economy may be achieved by rescaling 

20 the Sx, Sv, s z vectors by multiplying by a large integer, such as 2 16 , then changing from 
real to integer data types by truncating the fractional part. The result is that the real 
matrix- vector multiplication has been transformed into integer addition. To rescale the 
data back to the primary coordinate system, the transformed components are divided 
by the same factor which is done by bit shifting. 

25 From the geometry of the primary and the transformed secondary vol- 

ume data, the identity of the voxels that are actually in the field of view on the GUI, or 
that overlap in space are seen and can be processed. In Figure 10, some of the voxels 
on the periphery of the secondary volume data are outside the primary data and would 
not be used in the automated alignment method of the present invention. Only the use- 

30 ful voxels will be transformed, the remainder are not. 

Having now discussed the example with regard to scaling, interpola- 
tion, and transformation, MI maximization will be discussed. 
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Mutual information ("MI") maximization is now used to effect the opti- 
mal registration of the secondary volume data with respect to the primary volume data. 
Initially, there is consideration that primary and secondary volume data A and B, 
respectively, will have voxel values a and b, respectively. The voxel values are random 
variables that only assume finite integer value. These finite values will lie in the ranges 
of values 0 < a < N A - 1 and 0 < Z? < 7V B - 1 , where N A and N B are the numbers of val- 
ues observed in each data, respectively. The probability p(a) that a given A-voxel has 
the value a is defined as the number of voxels with value a, n(a), divided by the num- 
ber of all voxels in A. This is shown in Expression (37): 



" (37) 
X "(0 



i = o 



The set of all probabilities p(a) form the data set A and will be the probability distribu- 
tion for a. Moreover, it is also understood that for summation of the probabilities for all 
of the values of the voxels in A may be expressed as Expression (38): 



£ p{i) = 1 (38) 
i = o 

Therefore, to determine the p(a) from integer voxel data, a histogram of all the n(a) 
voxel value frequencies is constructed, then normalized according to Expression (37). 

An alignment of the primary and secondary data volumes, such that 
each voxel value a that is part of the primary data A(a e A) has associated with it a 
single secondary voxel b € B , will be based on the probability that the a y b combina- 
tion occurs together according to the joint probability p(a y b) which is set forth in 
30 Expression (39): 
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10 



p{a ,b) = (39) 

as Abe B 

where, 

n(a,b) = is the number of occurrences of the a y b combination. 

The probability p(a) of observing a voxel with value a is a measure of 
the uncertainty in predicting this observation. Therefore, for a single random variable 
A with a possible value a e A and a distribution p(a) at a given voxel in the primary 
data volume, the entropy (the measure of the uncertainty in the occurrence of a random 
variable) is set forth in Expression (40): 



15 



30 



H(A) = - £ />(a)log/>(a) < 4 °) 

a e A 



where, 

A = is a random variable that has the value a. 
logp(a)= is a logarithm in base 2. 
20 The entropy, as it is used here, is a function of the distribution p(a) and 

is maximized when the distribution of a is uniform, and is minimized if all of the vox- 
els in A have the same value. Referring to Expression (38), the entropy can be termed 
as a the average of l/logp(a). Therefore, for the two random variables A and B from the 
primary and secondary data volumes, the joint entropy is determined by Expression 
25 (41): 



H(A, B) = ^p(a 9 b)\ogp{a 9 b) (41) 

a = 0b = 0 

where, 

A = is a random variable representing the primary data. 
B = is a random variable representing the secondary data. 
a = is the actual value of random variable A. 
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b = is the actual value of random variable B. 

Given two random variables, one in the primary data volume repre- 
sented by A and one in the secondary data volume represented by /?, which have some 
unknown but determinable relationship, that relationship may described as will now be 
set forth. The relative entropy of the two random variables A and B is a measure of the 
distance between their two probability distributions. The MI is the amount of informa- 
tion one distribution contains about the other. The relative entropy between the two 
probability distributions, p{a), a e A and p(b), be B, is defined by Expression (42): 



D(p(a)\\p(b)) = £p(a)log^ (42) 

a e A 



The relative entropy, D, is viewed as a distance measure in that D(p(a)\\ p(b)) = 0 
15 when p(a) = p(b) , for all a,b, andZ>(/?(a) II p(b)) > 0 , for all a,b. 

MI for the purposes of the present invention is the relative entropy 
between the joint distribution and the product of the marginal distribution. Therefore, 
considering Expressions (41) and (42), MI is defined as Expression (43): 



10 



W) 0 =SXM» g ^ (43) 

a e Ab e B 

The MI, I(A;B), is the reduction in uncertainty in A due to the knowl- 
edge of B. Therefore, as shown in Figure 2 at graph 158, as the translations and rota- 
tions that have been previously described take place automatically, the MI increases 
until it reaches the state shown at 160. After this point, the alignment would only be 
minimally improved with additional computational time. The alignment process 
according to the present invention will normally range from about 30 seconds to 90 
seconds during autofusion. 

The terms and expressions that are employed herein are terms or 
description and not of limitation. There is no intention in the use of such terms and 
expressions of excluding the equivalents of the feature shown or described, or portions 
thereof, it being recognized that various modifications are possible within the scope of 
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TN THE CLAIMS : 

1. An autofusion method comprising the steps of: 

(a) generating a first image data volume of a structure using a first modal- 
ity; 

(b) generating an image data volume of the structure using a second modal- 
ity; 

(c) processing the first image data volume so that the first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so that the second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plane views of the first 
image data volume at two discrete locations on a display means; 

(f) selectively displaying at least two different plane views of the second 
image data volume at the two discrete locations on the display means, with the plane 
views of the second image data volume being of a same portion of the structure as the 
plane views of the frst image data volume; 

(g) scaling the plane views of the second image data volume to a scale of 
the plane views of the first image data volume; and 

(h) aligning the scaled plane views of the second image data volume with 
the plane views of the first image data volume including the substeps of, 

(1) translating the scaled plane views of the second image data volume 
to a position in each plane view in which predetermined points of the second image 
data volume approximate corresponding points in the first image data volume, 

(2) rotating the scaled and translated plane views of the second image 
data volume to a position in each view in which predetermined points of the second 
image data volume approximate corresponding points in the first image data volume, 
with the approximation at step (h)(2) being a closer approximation than the approxi- 
mation at step (h)(1), and 

(3) substantially simultaneously updating each plane view of the first 
image data volume with each selection movement with respect to the first image data 
volume, and 
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(4) substantially simultaneously updating each plane view of the second 
image data volume with each selection and alignment movement with respect to the 
second data volume. 

2. The method as recited in claim 1, wherein the structure is an anatomy. 

3. The method as recited in claim 2, wherein the first modality is com- 
puted tomography (CT) imaging. 

4. The method as recited in claim 2, wherein the second modality is mag- 
netic resonance imaging tomography imaging (MRI). 

5. The method as recited in claim 1, wherein step (e) includes selectively 
displaying at least three different plane views of the first image data volume at three 
discrete locations on a display means. 

6. The method as recited in claim 5, wherein step (f) includes selectively 
displaying at least three different plane views of the second image data volume at the 
three discrete locations on the display means, with the plane views of the second image 
data volume being of a same portion of the structure as the plane views of the frst 
image data volume. 

7. The method as recited in claim 6, wherein the three different plane 
views of the structure include an axial (transverse) view, coronal view, and sagittal 
view. 

8. An autofusion method comprising the steps of: 

(a) generating a first image data volume of a structure using a first modal- 
ity; 

(b) generating an image data volume of the structure using a second modal- 
ity; 

(c) processing the first image data volume so that the first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so that the second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plane views of the first 
image data volume at two discrete locations on a display means; 

(f) selectively displaying at least two different plane views of the second 
image data volume at the two discrete locations on the display means, with the plane 
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Automated Image Fusion/ 
Alignment System and 
Method 

5 

Field of the Invention 

The present invention relates to systems and methods for aligning 
image data volumes. More specifically, the present invention relates to aligning three- 
dimensional image data volumes. 

10 

Background of the Invention 

In the medical treatment area, it is usually necessary to utilize a number 
of modalities to view the internal anatomy of the person to be treated. These modalities 
include X-ray imaging, Magnetic Resonance Imaging ("MRI"), and computed tomog- 

15 raphy ("CT") imaging. There also are other modalities such as functional MRI 

("fMRI"), single photon emission computed tomography ("SPECT"), and positron 
emission tomography ("PET"), all of whose images contain physiologic or metabolic 
information depicting the actions of living tissue. 

It is known that each of modalities has certain strengths and weaknesses 

20 in the images that it produces. For example, X-radiography ("X-ray") imaging has high 
spatial and intensity resolutions, shows bony anatomy with high detail, and is rela- 
tively inexpensive to use; but X-ray also presents the viewer with complex two-dimen- 
sional ("2-D") views of superimposed anatomy. X-radiography also has difficulty 
resolving soft tissue features. MRI has the advantage of displaying three-dimensional 

25 ("3-D") images of soft tissues with high contrast and high spatial resolution, but does 
not image bone well. CT imagery, based on X-ray absorption, produces 3-D pictures of 
bony anatomy, and, increasingly, good definition of soft tissue, although MRI remains 
the preferred modality for viewing soft tissue. However, if the correct modalities are 
selected and aligned, the resulting combined image data will provide a more complete 

30 representation of the internal anatomy of the patient. Given this, the issue is to identify 
a system and method that will align image data from two modalities very accurately in 
a reasonable amount of time. 



l 
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Image alignment, which is part of the science of image fusion, has been 
used since at least World War II when time course aerial photography was used to sup- 
port strategic bombing. Blink comparator machines were used to superimpose two 
images taken at different times by displaying them to the viewer in rapid succession. 
5 As these images were being displayed in this manner, their features were matched to 
quantify the bombing damage. 

This same basic comparison technique has been used to superimpose 
images produced by remote sensors, for example aerial or satellite sensors, but with the 
refinement of using computer-based methods to align images produced at different 

10 times, or at the same time by different channels of the sensor. These images were digi- 
tal images. The techniques just described for image alignment in remote sensors were 
reported in Castleman, K.R., "Fundamentals of Digital Picture Processing/' Prentice- 
Hall, Englewood Cliffs, New Jersey, 1979, and Moik, J.G., "Digital Processing of 
Remotely Sensed Images," NASA SP-431 . Washington, D.C, 1980. 

15 The use of computers to effect image alignment became a necessity as 

the number of images produced by LANDS AT, SPOT, and other satellite systems rap- 
idly grew and there was the need to perform nonlinear warping transforms to match the 
images taken from different sky-to-ground perspectives. The use of computers also 
became a necessity to effect alignment of brain tissue images so that useful and effec- 

20 tive neurophysiological research could be conducted. This was reported in Hibbard, L, 
et al 9 Science . 236:1641-1646, 1987. Reviews of various computed alignment tech- 
niques have been reported in Brown, L.G., "A Survey of Image Registration Tech- 
niques," ACM Computing Surveys . 24: 325-376, 1992 and Van den Elsen, P. A., et al, 
"Medical Image Matching » A Review with Classification," IEEE Engineering in 

25 Medicine and Biology . March 1993, pp. 26-39. 

Image fusion of images from at least two modalities is currently being 
used in radiation onocology because it has been found to provide better tumor defini- 
tion, which was reported in Rosenman, J.G., et al, "Image Registration: An Essential 
Part of Radiation Therapy Treatment Planning," International Journal of Radiation 

30 Onocology. Biology, and Physics . 40: 197-205, 1998. It is anticipated that there will be 
increased development of software tools for the use in radiation treatment planning 
("RTP") to effect image alignment and display/contouring tools to process and use the 
fused images that are produced. 
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Image alignment methods generally fall in two categories: manual 
alignment by direct inspection of the images being aligned, and automatic alignment 
by computing a solution to a numerical problem with some type of computer program. 
Manual alignment ("MA") is carried out visually by matching corresponding features 
5 of two images. MA may be implemented in all RTP systems that offer or purport to 
offer image fusion. The mechanism that most frequently is used in MA is the visual 
placement of fiducial points or markers from which a transformation is derived, for 
example, by a least-square minimization of the differences between corresponding 
landmark points. A method of implementing MA that is not as frequently used is the 
10 visual matching of objects imbedded directly in the images being aligned. 

MA is usually carried out using a graphical user interface ("GUI"). In 
most cases, the GUI simultaneously displays the axial, sagittal, and coronal plane 
views of an area of interest. The GUI must provide efficient navigation of 3-D data, 
and precise localization of fiducial markers or other tools for alignment or measure- 
15 ment. Heretofore, commercial systems have used UNIX workstations or special-pur- 
pose computers to achieve the high computation and graphics throughput needed to 
interactively display large data volumes 

Automated image alignment ("ALA") methods involve obtaining a 
transformation through computation of the properties of the images to be registered. 
20 This may even take place through programmed computer actions without user inter- 
vention. Many of the more successful AIA methods are based on correlation and 
moment invariants matching. 

Correlation methods, more accurately, involve the cross-correlation or 
cross-covariance of image pixel intensities. These methods produce robust and accu- 
25 rate alignments of images. Methods of this type have been reported in Anuta, P.E., 

"Spatial Registration on Multispectral and Multitemporal Digital Imagery Using Fast 
Fourier Transform Techniques," IEEE Transactions on Geoscience Electronics . 8:353- 
368, 1970; Bamea, D.I., et al 7 "A Survey of Image Registration Techniques," IEEE 
Transactions on Computers . 21:179-186, 1972; and Pratt, W.K., "Correlation Tech- 
30 niques of Image registration," IEEE Transactions on Aerospace and Electronic Sys- 
tems, 10:353-358, 1974. 

According to most correlation methods, the location of the correlation 
peak will correspond directly to the translation needed to align images that already 
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have a correct rotational alignment. The rotational alignment may have been obtained 
by correlating the images after resampling them on a polar grid according to the 
method reported in Hibbard, L, et al, Science236: 1641-1646 , 1987. This rotational 
alignment method depends on the use of a fast Fourier transform. Correlation methods 

5 are accurate and robust to noise and differences in image content as long as the images 
are not too different. Accordingly, correlation methods are most often used for the 
alignment of images of the same kind. 

Moment invariants methods are computed methods for image registra- 
tion. A example of these methods is reported in Jain, A.K., "Fundamentals of Digital 

10 Picture Processing, Prentice-HalL Englewood Cliffs, N.J., 1989. This moment invari- 
ants method involves the use of the principal moments computed from a two-dimen- 
sional ("2-D") inertia matrix of some prominent object of the images. The principal 
moments correspond to a unique set of principal vectors. Image alignment is per- 
formed by transforming one image's principal vectors onto the other image's principal 

15 vectors. This usually is a fast calculation. However, the moment invariants method 

depends on the ability to efficiently extract the object or set of objects that serve as the 
moment fiducials. This method has considerable problems if the images are complex 
and have no dominant single feature for the alignment of images of the same kind or 
modality. 

20 Besides the moment invariants method just discussed, there are other 

alignment methods based on invariant properties. These other methods may use geo- 
metric invariants, and minima/maxima of curvature and other extrema. At least one of 
these methods is reported in Lavellee, S., "Registration for Computer Integrated Sur- 
gery: Methodology, State of Art, in Taylor, R., et al, Computer In tegrated Surgery, 

25 MIT Press, Cambridge, MA, 1996. The characterization of these methods are that they 
are fast but susceptible to noise and inter-subject differences. These systems, however, 
only use a small portion of all of the available information from each image set to com- 
pute the registration, which has some effect on the accuracy. 

In a practical sense, the automated alignment of CT and MRI images of 

30 the head was accomplished in the late 1980's and early 1990's. The methods that were 
used at that time gained some level of use but they had drawbacks. These methods 
were reported in Pelizzari, C.A. et al, "Accurate Three-dimensional Registration of 
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CT, PET, and/or MR Images of the Brain/' Journal of Computer Assisted Tomography , 
13:20-26, 1989. 

The method described in Pelizzari, et al suggests representing the sur- 
face of the brain in CT and MRI by stacks of contours of the cortical surface and mini- 
5 mizing the least-squares differences between neighboring points in the two surfaces to 
determine the alignment transform. This method was later refined by sub-sampling the 
surface points and then applying simple optimization to determine the transformation. 
This late refinement was reported in Van Herk, M. and Kooy, ELM., "Automatic Three- 
Dimensional Correlation of CT-CT, CTMRI, and CT-SPECT Using Chamfer Match- 

10 ing," Medical Physics . 21:1163-1178, 1994. It was found that the original and refined 
methods were useful only for the head, and both methods required that the cortical sur- 
face be manually contoured first. 

Another automatic method that has been suggested for the registration 
of images is based on the maximization of mutual information and this method was 

15 first reported on in 1995. This method has its genesis in information theory in which 
the relatedness of one random variable for another is based on the measure of the vari- 
ables' entropies, which is referred to as mutual information ("MI")- (See, Cover, T. and 
Thomas, J, Elements of Information Theory . John Wiley and Sons, New York, 1991). 
Thus, for two images, the MI is small if the images are unrelated, or related but unreg- 

20 istered. If the images registration is improved, their MI increases and is maximized 

when the images are geometrically aligned. The MI is different from correlation in that 
systematic differences between images which would confound correlation actually 
strengthen the alignment by MI. MI is frequently used for two- and three-dimensional 
alignment of multimodal imagery in medical science as reported in Wells, W.M., et al, 

25 "Lecture Notes on Computer Science," Vol. 1496, Springer- Verlag, New York, 1998. 

The use of MI has also been the subject of a number of studies. For 
example, it has been used for aligning multimodal MRI and MRI with CT as reported 
in Viola, P. and Wells, W.M., "Alignment by Maximization of Mutual Information," 
Proc. of the Vth Int'L Conf. on Computer Vision . Cambridge, MA, June 1995, pp. 16- 

30 23; Collignon, A., et al , "Automated Multimodality Medical Image Registration 

Using Information Theory," in Bizais, Y, etal, Proc. of the XVth Int'L Conf. on Com- 
puter Vision. Virtual reality, and Robotics in Medicine (CVRMed '951 , Vol 905, 
Springer- Verlag, Lecture Notes in Computer Science . Nice, France, April 1995, pp. 
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195-204; Meas, R, et al, "Multi-Modality Image Registration by Maximization of 
Mutual Information/ 7 IEEE Workshop on Mathematical Methods in Biomedical Image 
Analysis . San Francisco, CA, June 21-22, 1996, pp. 14-22; and Studholme, C, et a/., 
"Automated 3-D Registration of MR and CT Images of the Head," Medical Image 
5 Analysis . 1:163-175, 1996. Among other things, the issue with MI, however, is the 
ability to perform it in a time effective manner. 

In 1997, the Vanderbilt Retrospective Brain Registration study was con- 
ducted. This study was directed to computed alignment methods with the goal of 
assessing the error of each of the standard group of head data sets comprising CT+MRI 
10 or PET+MRI data sets in which "gold standard" alignments (alignments by which the 
others were judged) were measured using bone-implanted markers. In the study, evi- 
dence of the markers were removed from the images before being sent to the partici- 
pating laboratories. The techniques used by the testing laboratories included variations 
of the surface matching, correlation, and ML The errors in respective results varied 
15 greatly. There were at least fourteen methods used to align CT and MRI images. Two 
of the laboratories that used MI achieved the lowest total median error over the spec- 
trum of MRI pulse-sequence modalities. The third best results were achieved by a cor- 
relation method. In a comparison of alignments using PET and MRI, the first and 
second lowest total median errors were achieved by a single correlation method fol- 
20 lowed by two MI methods. In summary the MI and correlation methods produced the 
lowest errors and did not require brain contouring, scalp removal, or other interactive 
activities. Although these methods showed some success, it is believed that the time to 
solution could be very much improved without sacrificing accuracy. 

The present invention provides a system and method that includes an 
25 automated alignment method that is accurate but much faster that the prior art systems 
and methods; and the system and method of the present invention includes an interac- 
tive alignment method that achieves speeds and displayed detail equal to, or exceeding, 
the highest performance level of the prior art, without the need of specially configured 
or extremely powerful high performance workstations. 



30 



Summary Of The Invention 

The present invention is a system and method for autofusion and in par- 
ticular autofusion of three-dimensional ("3-D") image data volumes. According to the 
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system and method of the present invention, registration is effected by simultaneously 
displaying on a GUI at least, but preferably, two 3-D, image data volumes, and one of 
the 3-D image data volumes is held fixed while the other may be scaled, rotated, and 
translated to align homologous anatomic features. The system and method will simul- 
5 taneously display the axial, sagittal, and coronal plane views of a region of interest. 
Preferably, these plane views will be data volumes that are designed and configured for 
a system user to "drag" or "position" one volume to a new position in one plane view 
which will also simultaneously update the data volumes as displayed in the other two 
plane views with no apparent lag time. The system and method of the present invention 
10 also includes automated alignment computation based on mutual information ("MI") 
maximization. The system and method of the present invention using MI maximization 
aligns 3-D more efficiently and faster than prior image alignment methods. 

According to the system and method of the present invention, the dis- 
play and automated alignment referred to above have that advantage of achieving high 
15 speed updates of the data volumes on the display while at the same time displaying the 
data in full resolution. The system and method accomplish this following specific strat- 
egies. First, the composite transformations of two independent data volumes are repre- 
sented as a pair of homogeneous matrices whose product directly generates the point in 
one system homologous to a given point in the other system. Second, the volume data 
20 transformation is recast from matrix multiplication of real- valued vectors to the addi- 
tion of integer vectors that have components which approximate the real- valued vector 
components. Third, there is a determination of that portion of a row or column of data 
that will be visible for display or lie in the space of the fixed data, for automated align- 
ment, before performing the transformation and resampling. 
25 Autofusion according to the present invention is characterized as the 

alignment and combined use of multiple kinds of images of a region of interest. This 
combined use of the multiple kinds of images permits the extraction of information 
about the region of interest that would be unavailable from any one of the images. As 
such, the registration or alignment process is one that spatially matches homologous 
30 objects of various images and a process for extracting synergistically derived informa- 
tion. 

Autofusion of the present invention is carried out with initial manual 
selection of images to be aligned, followed by either manual alignment using the GUI 
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or by automated alignment. The alignment methods may be used in either order, or 
repeatedly in any order, without restriction. More specifically, this involves the interac- 
tive matching of image objects using a graphical user interface ("GUI") and an auto- 
mated method to compute the estimated alignment of the two images without user 
5 interaction. These combined actions according to the present invention are comple- 
mentary and the alignment that is produced is more accurate and time effective than 
prior methods. 

The manual selection and initial superimposition of the two images is 
done with the GUI. The automated alignment uses a computed MI maximization 

10 method that automatically scales, and finds the translations and rotations that will pro- 
duce the best correspondence of the images. At least one embodiment of the present 
invention contemplates that the geometric transformation that is applied to the movable 
data volume is for linear transformations involving six degrees of freedom: three trans- 
lations and three rotations. 

15 The system and method of the present invention will described in 

greater detail in the remainder of the specification referring to the drawings. 

Brief Description Of The Drawings 

Figure 1 is a perspective view of the GUI employing the present inven- 
2Q tion that shows the unaligned CT and MRI images of the same patient in the trans- 
verse, coronal and sagittal planes. 

Figure 2 is a perspective view of the GUI employing the present inven- 
tion that shows the CT and MRI images in the transverse, coronal and sagittal planes 
from Figure 1 aligned. 

25 Figure 3 is a representative drawing of a CT stack containing an image 

data volume. 

Figure 4 is a representative drawing of a MRI stack containing an image 

data volume. 

Figure 5 is a representative drawing of the data volume coordinate sys- 
3 q tern for use in the present invention that shows the transverse (axial), coronal and sagit- 
tal planes. 

Figure 6 is an example of a primary and secondary data volumes having 
different scales. 
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Figure 7 is an example of primary and secondary data volumes having 
scaling and translation differences- 
Figure 8 is an example of primary and secondary data volumes having 
scaling, translation and rotation differences. 
5 Figure 9 is an example of data interpolation for the secondary data vol- 

ume on the primary data volume. 

Figure 10 is an example of the transformation of a secondary data vol- 
ume onto a primary data volume by iterated, integer- vector addition. 

10 Detailed Description Of The Drawings 

The present invention is a system and method for autofusion of three- 
dimensional ("3-D") image data volumes. According to the system and method of the 
present invention, autofusion is effected by simultaneously displaying on a GUI at 
least, but preferably, two 3-D, image data volumes, and one of the 3-D image data vol- 

15 umes is held fixed while the other may be scaled, rotated, and translated to align 

homologous anatomic features. The system and method will simultaneously display 
the axial, sagittal, and coronal plane views of a region of interest. Preferably, these 
plane views will display the voxel intensities on 2-D planes within the data volumes, 
whose locations within the data volume are specified by the locations of straight-line 

2Q cursors in the GUI axial, coronal, and sagittal windows, such as "dragging" a cursor in 
any window specifies a new location for one of the plane views, and that plane view is 
immediately updated with no lag apparent to the user. In a second method of operation, 
one of the two data volumes may itself be "dragged" with a cursor to effect a rotation 
or translation with respect to the fixed data volume. This will cause all three views, 

25 axial, coronal, and sagittal, to be re-displayed with no lag apparent to the user. In a 
third method of operation, the movable data volume may "jump" and "twist" in real- 
time response to the latest, "best," alignment obtained during the maximization of 
mutual information determination. Autofusion of the present invention also includes 
automated alignment computation based on mutual information ("MI") maximization. 

2Q The system and method of the present invention using MI maximization of aligns 3-D 
more efficiently and faster that prior alignment methods. 

Autofusion of the present invention is carried out with an initial manual 
selection of images to be aligned, followed by either manual alignment using the GUI 



9 



WO 01/022363 PCT/US00/20291 

or by automated alignment. The alignment methods may be used in either order, or 
repeatedly in any order, without restriction. That is, first, there is interactive matching 
of image objects using a graphical user interface ("GUI") and then there is an auto- 
mated method to compute the estimated alignment of the two images without user 
5 interaction. This combination produces more accurate alignments than prior methods. 

Autofusion, according to the present invention, involves the registration 
of different image volumes by interactive matching of image objects using a GUI and 
automated computation of alignment. The use of the GUI provides the system user 
with a three views of the 3-D data volumes at issue, and permits the system user to 

10 translate and rotate the image data of one modality with respect to the other. For exam- 
ple, the desire may be to autofuse CT and MRI studies. For the puiposes here each 
study is in the form of a data volume. The CT image data will be in a fixed position of 
the GUI while the MRI image data may be rotated and translated with respect to the 
CT image data. In this case, the CT image data is referred to as the primary data set and 

15 the MRI image data is referred to as the secondary data set. The primary and secondary 
data sets will have window and level controls on the GUI. There also will be a slider 
control at the GUI for controlling the intensity of the primary and secondary data sets 
in the windows. 

The display (through the GUI) and automated alignment achieve both 
20 high speed updates of the GUI while displaying the data at full resolution. Autofusion 
of the present invention depends at least in part on (i) representing the composite trans- 
formations of the two independent data volumes as a pair of homogeneous matrices 
whose product, with that of the transformation, directly generates the point in second- 
ary system homologous to a given point in the primary system; (ii) the volume data 
25 transformation is recast from matrix multiplication of real-valued vectors to the addi- 
tion of integer vectors whose components approximate the real vector components; 
and (Hi) determining the portion of a row or column of data which will be visible on 
the display or will lie in the space of the fixed data (for automated alignment) before 
performing the transformation and resampling so that needless transformation are not 
30 carried out. 

In the practical application of the present invention, autofusion will 
align two 3-D image data volumes for use in RTR Typically, this will consist of align- 
ing one X-ray CT study set and one MRI study set. The alignment of the images of 
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these two modalities is to augment the CT anatomic information with greater soft tis- 
sue detail that is seen in MRI imagery: the contours of soft tissue organs and tumors 
after better shown in MRI than in CT. If the two modalities are accurately superim- 
posed, the MRI images, where needed, can substitute for what is not shown or not 
5 shown well in the CT images. Autofusion of the present invention may also employ 
specific tissue functional information from nuclear medicine imagery (SPECT, PET) 
or functional MRI imaging (fMRI) which may be helpful to obtain more accurate and 
effective treatment. 

Referring to Figure 1, generally at 100, a representative GUI is shown 

10 for use in the present invention. Then present invention is preferably for aligning a pri- 
mary (CT) and secondary (MRI) image data to aid in contouring tissue, although many 
other uses are contemplated by the inventor. 

In Figure 1, the transverse or axial plane view of the primary and sec- 
ondary data sets is shown at window 102. The primary data set is shown at 104 and the 

15 secondary is shown at 106. The coronal plane view of the primary and secondary data 
sets is shown at window 108. The primary data set is shown at 110 and the secondary 
is shown at 112. The sagittal plane view of the primary and secondary data sets is 
shown at window 114. The primary data set is shown at 116 and the secondary is 
shown at 118. In each of the windows, the primary and secondary data sets have con- 

20 trasting colors or one set could be colored and the other black and white. However, all 
that is necessary is that the two sets be colored so that they may be distinguishable. 
This is necessary so the user can tell how the secondary is being rotated and translated 
with respect to primary. As is readily seen by reviewing windows 102, 108, and 114 
and the contrasting colors of the two data sets, the primary and secondary data sets are 

25 unaligned. The blank area 132 will remain this way until MI maximization is per- 
formed at which time the MI levels will be displayed as will be discussed subse- 
quently. 

Again referring to Figure 1, the primary window control is shown at 
122 and primary level control is shown at 124. Primary window control 122 is used to 
30 select the window from the primary data set that will be displayed in windows 102, 
108, and 114. As such, the number 1301 indicates that the primary data in an interval 
1301 intensity units wide, centered on the level value of 164, is displayed. Primary 
level control 164 is used to set the intensity level for the primary data set displayed in 
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windows 102, 108, and 114. Thus, the number 164 indicates that the primary data in an 
interval 1301 intensity units wide, centered on the level value of 164, is displayed. 

In a similar fashion, the secondary window control is shown at 126 and 
secondary level control is shown at 128. Secondary window control 126 is used to 
5 select the window from the secondary data set that will be displayed in windows 102, 
108, and 114. As such, the number 100 indicates that the secondary data in an interval 
100 intensity units wide, centered on the level value of 50, is displayed. Secondary 
level control 128 is used to set the intensity level for the secondary data set displayed 
in windows 102, 108, and 114. Thus, the number 50 indicates that the secondary data 

10 in an interval 100 intensity units wide, centered on the level value of 50, is displayed. 

At the bottom of GUI 100 is a slider with the labels "Primary" and 
"Secondary" at respective ends. Movement of the control will increase and decrease 
the intensity of the secondary with respect to the primary. Accordingly, if the user 
wants to emphasize the primary data set more than the secondary data set, he or she 

15 will move the control closer to the "Primary" label; and if the user wants to emphasize 
the secondary data set more than the primary data set, he or she will move the control 
closer to the "Secondary" label. The use of this control is important in selecting and 
aligning the primary and secondary data sets on the GUI. 

Referring to Figure 1, windows 102, 108, and 114 each have a circle 

20 and two line-cursors. More specifically, window 102 has circle 134, vertical line-cursor 
136, and horizontal line-cursor 138; window 108 has circle 140, vertical line-cursor 
142, and horizontal line-cursor 144; and window 114 has circle 146, vertical line-cur- 
sor 148, and horizontal line-cursor 150. The two line-cursors in each window depict 
the locations of the other two plane views in data volume. Referring to window 102, 

25 horizontal line-cursor 138 shows the location of the coronal section shown in window 
108 and vertical line-cursor 136 shows the location of the sagittal section shown in 
window 114. Similarly, referring to window 108, horizontal line-cursor 144 shows the 
location of the transverse or axial section shown in window 102 and vertical line-cur- 
sor 142 shows the location of the sagittal section shown in window 114. Finally, refer- 

30 ring to window 114, horizontal line-cursor 150 shows the location of the transverse or 
axial section shown in window 102 and vertical line-cursor 148 shows the location of 
the coronal section shown in window 108. 
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Movement of vertical line-cursor 136 in window 102 will cause the sag- 
ittal plane view in window 114 to pan and movement of horizontal line-cursor 138 in 
window 102 will cause the coronal plane view in window 108 to pan. Likewise, move- 
ment of vertical line-cursor 142 in window 108 will cause the sagittal plane view in 
5 window 114 to pan and movement of horizontal line-cursor 144 in window 108 will 
cause the transverse or axial plane view in window 102 to pan. Lastly, movement of 
vertical line-cursor 148 in window 114 will cause the coronal plane view in window 
108 to pan and movement of horizontal line-cursor 150 in window 102 will cause the 
transverse or axial plane view in window 102 to pan. The movement of these line-cur- 

10 sors permit the user to navigate the data volumes that are shown, through the simulta- 
neous updating of the affected views with each movement of the line cursors. 

Circles 134, 140, and 146 in windows 102, 108, and 114, respectively, 
indicate the areas where the user may engage a particular window view to manually 
rotate or translate the secondary image. Thus, the GUI cursor, which is to be distin- 

15 guished from the vertical and horizontal line-cursors, is positioned within and without 
the a circle to effect rotating and translating the MRI data set (the secondary data set). 
If the GUI cursor is placed within the circle and engaged, it may be used to translate 
the secondary image in two directions in the plane of the window. This translation will 
be evidenced in the other two plane views simultaneously. If the GUI cursor is placed 

20 outside of the circle and engaged, it may be used to rotate the secondary image about 
the center of the plane of the window. The rotation will be evidenced in the other two 
plane views simultaneously. 

When two data volumes for registration are selected, the first action is 
the manual manipulation of the unaligned views. This will involve the selection of the 

25 proper primary and secondary window views, and the desired intensity levels for these 
two window views. The relative intensity between the primary and secondary images 
may be controlled by slider control 130. Once the proper views and levels are selected, 
the line-cursors of the respective windows are used to obtain suitably informative 
views of the primary and secondary data sets in each of the windows. Following this, 

30 the GUI cursor is positioned within and without the circles of each window to translate 
and rotate the secondary images for alignment of the secondary (movable) image with 
the primary (fixed) image. 
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Automated alignment, using MI maximization, may be performed 
before, after, or instead of, manual alignment. Automated alignment is initiated by 
activation of automated alignment control shown at 152 in Figure 1. The activation of 
the automated alignment control will cause the improved alignment of the secondary 
5 image on the primary image. 

Figure 2, shown generally at 200, shows windows 102, 108, and 114 
with the primary and secondary image aligned or registered. In Figure 2, the elements 
that are the same as those in Figure 1 have the same reference numbers. The alignment 
of the primary and secondary images in Figure 2 is the result of automated alignment 

10 techniques according to the present invention. It is to be noted that the alignment that 
takes place is with respect to all three of the plane views, transverse (or axial), coronal, 
and sagittal, simultaneously. 

The progress and accuracy of the automated alignment method is 
shown in window 156 which contains graph 158 that shows a plot of the value of the 

15 mutual information versus iteration number. The final MI maximization value is dis- 
played at 160 of window 156. This later value is a very accurate registration of the pri- 
mary and secondary images so that useful information from each image may be use, 
for example, in RTR As automated alignment is being performed, the image displays 
are updated to reflect the incremental improvements in the alignment. The method by 

20 which automated alignment of the primary and secondary image is carried out will 
now be describe 

Both the display and automated alignment of the present invention 
require rapid determination of CT or MRI intensities at arbitrary locations in the 3-D 
primary and secondary data volumes. According to the present invention, translation 

25 and rotation of a data volume in a single window will result in the computation of the 
position in the corresponding plane in the data volume. Correspondingly, the intensi- 
ties are determined for that plane position. Since digital image data occurs only at a 
finite number of rows, columns, and slices, the viewing lines located between the rows, 
columns, or slices must estimate the intensities. Likewise, the automated alignment 

30 performs a voxel-by-voxel comparison of the overlapping parts of the primary and sec- 
ondary data of the data volumes. This requires comparing primary voxels at integer 
row, column, and slice coordinates with transformed secondary voxels at non-integer 
locations in the primary coordinate system. Accordingly, the present invention has the 
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ability to determine transformed locations and intensities at new locations very rapidly, 
preferably at a rate of several million or greater per second, to have the GUI views 
updated without visible lagging. This is accomplished by casting the geometric trans- 
formations in an efficient and compact representation and substituting for the time 

5 costly multiplication of real (decimal) numbers with serial addition of integers. More- 
over, according to the present invention, integer rescaling of the transformations of the 
secondary data volume is done by bit shifting which facilitates faster results. 

To improve the time efficiency of the method of the present invention, 
there is a determination whether the volume data to be transformed remains in the 

10 domain of the display or the overlap volume. The present invention only performs 

transformations on those voxels which contribute usefully to these functions. Further, 
to increase the speed of the results, scan line visibility checking may be accomplished 
before transformation takes place since the geometries of these data and the display 
coordinate system are known so transformation is not necessary. 

15 As background before discussing the autofusion of the present inven- 

tion, the CT and MRI data sets that preferably are used, for example, in RTP are sets of 
samples of a patient's anatomy at regular intervals on a 3-D rectangular grid. CT 
images represent sections of the anatomy reconstructed from absorption measurements 
obtained as projections through the patient. The reconstruction plane pictures form the 

20 transverse or axial plane view of the patient. Normally, CT studies have a spatial reso- 
lution of 0.7 to about 1 mm/voxel (volume element) in the plane of the reconstructed 
section images. Typically, CT images have dimensions of 512 x 512 voxels in the sec- 
tion planes and may have 40 to greater than 100 sections for a study. 

MRI images are reconstructed from volumetric magnetization measure - 

25 ments. These images are reconstructed along planes normal to the axis of the patient, 
so, normally, the images set is formed in the transverse or axial orientation and, there- 
fore, provide comparisons with CT imagery of the same patient. MRI spatial resolu- 
tions are approximately 1.5 to 3 mm/voxel in the plane of the reconstructed images. 
These MRI images normally have dimensions of 256 x 256 voxels in the section 

30 planes. Typically, MRI studies have 30 to greater than 100 sections. 

The most common presentation of both CT and MRI images is a stack 
of images showing sections of the patient's anatomy in planes perpendicular to the 
scanner's axis, and the patient's inferior-superior axis. A representative CT stack 
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which includes a data volume is shown in Figure 3 generally at 270 and a representa- 
tive MRI stack which includes a data volume is shown in Figure 4 generally at 272. 
The images along the patient's inferior-superior axis are referred to as transaxial (or 
axial or transverse). Images in planes perpendicular to the anterior-posterior direction 
5 are coronal and sagittal images, and are in planes perpendicular to the medial-lateral 
direction. For description purposes only, the following example of CT and MRI images 
data sets is provided for a patient that will be using RTP: superior -- toward the head; 
inferior - toward the feet; anterior - toward the top or front of the body or toward the 
face; posterior - toward the back of the body or head; medial - toward the center of 
10 the body or head; and lateral - toward the exterior side of the body or head. 

Assuming a supine position of a patient when a CT or MRI scan is per- 
formed to produce the data stacks shown in Figures 3 and 4, the data volume coordi- 
nate system showing the transverse or axial, sagittal, and coronal planes, along with 
the x, y, and z axes is shown in Figure 5 generally at 274. In Figure 5, the axial plane is 
15 shown at 275, and the sagittal plane at 276, the coronal plane at 277. The x, y, and z 
axes are shown generally at 278. 

Image data consists of scalar intensities at regular intervals on a 3-D 
right rectangular grid with orthogonal coordinate axes (x,y,z). The z axis is parallel to 
the inferior-superior axis or perpendicular to the transverse or axial image sections. 
20 The x axis is parallel to the anterior-posterior axis and perpendicular to the coronal 

image sections. Finally, the y axis is parallel to the medial-lateral axis and perpendicu- 
lar to the sagittal image sections. The x and y axes lie in the transverse or axial image 
plane with the origin at the upper left of the image. The x axis is parallel to the medial- 
lateral axis and perpendicular to the sagittal image sections. The y-axis is parallel to 
25 the anterior-posterior axis, and perpendicular to the coronal image sections. 

Preferably, the image data that is being processed according to the 
present invention is digital data. As such, the CT and MRI data for the data volumes 
will be an array of numbers with each number representing a measurement or sample 
of a tissue property at a point in space. These data points are taken to exist at the cen- 
30 ters of voxels which are rectangular volume with a length, width, and height equal to 
the characteristic spacing between data. The geometry of the data is consistent with a 
Cartesian coordinate system that has an origin and axes at right angles to each other. 
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The automated alignment portion of the autofusion method of the 
present invention is for automatically performing secondary image transformation, 
which in this case is meant to mean three rotations and three translations of the second- 
ary image with respect to the primary image that will maximally superimpose homolo- 
5 gous anatomical points. This optimal transformation, F* corresponds to the maximum 
value of the mutual information represented by I(p,F(s)) which relates that primary 
image data to the transformed secondary image data according to the Expression (1): 



10 F* = maxT(p, F(s)) (1) 

F 

where, 

/ = is the real value of the MI. 

p = is an array of the primary intensities. 

F(s) = is the array of the secondary intensities transformed by F. 

In most cases, the real-valued rotations and translations relocate the 
integer-indexed secondary voxels to non-integer locations. Accordingly, the secondary 
image data must be interpolated to obtain secondary voxel intensities at the primary 
voxel locations. These interpolated F(s) intensities are to form the marginal and joint 

20 

probability distributions that are needed to determine the ML 

The determination of the MI maximization is dependent on an evalua- 
tion of the transformations, interpolation, and probability distributions underlying 
Expression (1). These transformations, interpolation, and probability distributions will 
now be discussed. 

25 

First the coordinate transformations must be considered. A transformed 
secondary data voxel may be represented by s t . This value is transformed from the ini- 
tial state of a secondary data voxel which is represented by s=(s x , s y s z ). The transfor- 
mation is accomplished by a combination of scaling, translation, and rotation 
operations. This transformation is represented by Expression (2): 
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s t = R(y-s-t) (2) 



is (v x , v y , v z ), the vector of the primary to secondary axis scale 

is (cp x , (py, <p z ), the matrix of rotations about the three axes. 
is (t x , ty, t z ), the translation vector. 

is (s^ s r >s z ), the initial secondary data voxel coordinates before 
scaling, translation, or rotation. 

The representative relationship of the primary and secondary data of the 
15 data volumes prior to scaling or translation is shown in Figure 6, generally at 300. In 
Figure 6, the primary and secondary data volumes are shown with the same origin at 
302 so that the differences may be readily seen. The primary data is shown at 304 and 
the secondary data at 306. It also is readily seen that the x and y spacing for the primary 
and secondary at 308,310 and 312,314, respectively is different. The differences 
20 between the primary data volume and the secondary data volume in Figure 6 is due to 
scaling differences. As such, before any further alignment can be made, it also is nec- 
essary for the secondary data volume to be scaled to primary data volume. 

The vector v x , v r v z is known from scanner calibrations of the image 
data and it is considered fixed. Therefore, it is only necessary to vary the translations 
25 and rotations to obtain an optimal alignment or registration once scaling has taken 
place. 

The necessary coordinate transformations may be performed using 
matrix multiplication 

as set forth below. In this case, these transformations may be presented by matrix oper- 
30 ations in homogeneous coordinates in which vector translation, rotation, scaling, and 
skewing operations are combined in one notation. In the three dimensions of the data 
volume, vectors are represented as 4 x 1 arrays according to Expression (3): 
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where, 



v 

ratios. 

R 

t 
s 
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(s x ,s y ,s z , if = 



(3) 



10 



where, 

s = the initial vector for the secondary data voxel. 

T = transpose of the vector. 

Considering Figure 6, in order to compare the intensity at a point a of 
the primary image data with the same point in the secondary image data requires inter- 
polating the secondary image data values. The secondary distances D s are scaled to the 
primary distances D p , and this may represented by Expression (4): 



15 



x 0 



(4) 



The corresponding scaling components are set forth in Expression (5): 



20 




(5) 



The corresponding transformation is shown in Expression 6 which will be discussed 
subsequently. 

The linear and affine transformations, such translations, rotations, scal- 
ing, and skewing, are represented by 4 x 4 matrices. More particularly with regard to 
scaling, this is achieved by the multiplication of the vector of scale factors v in a 4 x 4 
matrix (with the components of the vector v on the diagonal) with the 4 x 1 matrix 
shown in Expression (3). This is shown in Expression (6): 
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VS = 





0 


0 


0 




s x 




V x* x 


0 


v y 


0 


0 




s y 






0 


0 


v z 


0 




s z 




v z S z 


0 


0 


0 


1 




x 




1 



= s 



(6) 



10 



15 



20 



The determination of s t must made repeatedly for the plurality of sec- 
ondary data voxels. As such, for efficiency, it is convenient to pre-multiply s times v as 
shown in Expression (6). The determination of s' according to Expression (6) serves 
as a resampling equivalent for the secondary image data along all three axes so that the 
same grid spacing for the secondary image data is achieved as is present in the primary 
image data. 

Referring to Figure 7, generally at 320, shows a situation in which the 
secondary data volume differs from the primary data volume by scale factors and a 

■> 

translation T. The translation difference is the sum of the component translations 
along x and y, and represented at 338 in Figure 7. 

Taking this into account, the determination of ^may be based on one 
translation and one rotation of the secondary image data about each of the three axes as 
is seen by Expression (7): 



R(s'-t) 



(7) 



25 where, 



30 



s 
t 



R = is (cp^, (p y , (p z ), the matrix of rotations about the three axes. 

= the product of v and s. 
= is (t x , t y t z ), the translation vector. 
The translation of s' by vector t=(t x , t y t z ) is represented by the multi- 
plication of this vector by the 4x4 matrix T with the translation components in the 
fourth column of the matrix, as set forth in Expression (8): 
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s ' - t = Ts = 



10 0^ 










0 10^ 










0 0 1 t z 




*z 






0 0 0 1_ 




_1_ 




1 



Next the rotations of the secondary image data must be considered. 
Referring to Figure 8, generally at 350, the combined offsets due to scaling, translation 

by T at 356, and rotation by angle 0 at 358 is shown. In Figure 8, the primary volume 
data is shown at 352 and the secondary volume data is shown at 354. The rotations of 
the secondary image data by the angles q x , <p r (p z about the three coordinate axes is 
according to Expressions (9), (10), and (11): 



10 0 0 
0 cos^ -sin § x 0 

0 sin$ x cos$ x 0 

0 0 0 1 



(9) 



cos$ y 0 sin$ y 0 
0 10 0 

-sinc^ 0 cos 4^ 0 
0 0 0 1 



(10) 



cos<j> z -sin<]) z 0 0 
sin(j> z cos$ z 0 0 
1 0 
0 1 



0 0 
0 0 



(11) 



The rotation of secondary image data, s' , about the z-axis, which will 
be in the xy plane, is represented by Expression (12): 
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cos(j) z -sin(|) z 0 0 
sin$ z cos()) z 0 0 
0 0 10 
0 0 0 1_ 



^cos<j> z -s y sin<() z 



(12) 



10 



This same type of matrix multiplication is used for the other two axes, R($ x )s and 

R(Q>y)S. 

The combinations of the coordinate transformations may be expressed 
as the products of the individual transformation matrices. The combined transforma- 
tion equivalent to the translation followed by rotation of the secondary image data 
about the z axis is set forth in Expression (13): 
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s tr = R($ z )Ts 



cos<j> z — sin^ 

sin<|) z cos<j), 

0 0 
0 0 



'z 

'z 
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lOOt, 




s x 
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0 1 0 ^ 




Sy 
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0 0 1 t z 




s z 




0 0 0 1 




1 
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-sin<|> z 0 0 



cos(j> z 

sin(j> z cos(|> z 0 0 

0 0 10 

0 0 0 1 



S y-ty 

s z-h 



(13) 
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(s x - t x ) cos(|> z -(5 3 , - t y )sm$ z 
(s x - ^)sin<j) z + (s y - t y )cos$ z 



Noting the determination of s tr above, the computations necessary for 
homogeneous coordinates may be inefficient unless specially designed hardware sys- 
tems are used since there are a large number of multiplications by zero and one. As 
such the present invention utilizes an approach that will produce accurate results with- 
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out the need for the level of complex computations that have been a problem in the past 
to perform in a time-efficient manner. This approach will now be described. 

The rotations of the secondary image data are all basically 2-D. Thus, 
they may be restated as 2 x 2 matrices containing only the sin and cos terms of Expres- 

5 sions (9), (10), and (11) multiplied by the appropriate combinations of x, y, z taken two 
at a time. These rotations also must be performed with reference to an origin or rota- 
tion center. Preferably, the rotation center is not the same point as the primary and sec- 
ondary data volume origins. 

As stated, the origins for the primary and secondary data volumes are 

10 at one corner of the rectangular volumes. The rotations of the secondary image data 
about its origin will introduce large and unpredictable effects in the attempt to find 
optimal alignment of the common features of the primary and secondary image data. 
As such, according to the present invention, the rotation center is set at the center of the 
secondary data volume so that the rotations will leave the secondary volume data 

15 largely superimposed on the primary volume data. To carry out this action, there are 
two translations that are of concern. The first is to translate the secondary data volume 
center to the coordinate system origin, which preferably will be defined with respect to 
the primary data volume. The second is to translate the secondary data volume back to 
its original position. The first translation is represented by Expressions (14) and (15): 

20 
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(14) 
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is the transformed secondary data voxel, 
is the matrix of the translation of the secondary volume data 
with its center at (c x , c y c z ) to the coordinate system origin. 

is the inverse translation of the C matrix. 
is (<p x , <p 3 „ <p z ), the matrix of rotations about the three axes, 
the product of v and s. 
is the transpose of the vector. 



30 



where, 

C 



c 1 

R 



T 
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Now expanding the rotation term, R, to embrace the rotations in the three axes, the 
total transformation is represented by Expression (15): 



s t = C~ l R z R y R x CTs 



(15) 



10 



As previously stated, the rotations about the x, y, and z axes involve changes 
only to the two coordinates in the plane to that axis. Therefore, with regard to each 
axis, the information being considered is 2-D rather than 3-D. As a result, this will 
reduce the 4 x 4 matrix multiplications, considering Expressions (9), (10), (11), and 
(12), to a series of 2 x 2 matrix multiplications, which is shown in Expressions (16), 
(17), and (18): 



15 



cos^ -sin^ 
sin^ cosc^ 



, u = s y , v = s 



(16) 



20 



RyS' 



cos$ y sin^ 
-sin<j> y cos<(> yJ 



, U = S „, V = s 



(17) 



25 



R z s' 



cos<j> z -sin<|> z 
sin()> z cos<|> z 



, u = s x , v = s 



(18) 



where, 



30 



Ry 

Rz 

U 
V 



is the rotation about the x axis in the y and z plane, 
is the rotation about the y axis in the x and z plane, 
is the rotation about the z axis in the x and y plane, 
is the first component, 
is the second component. 
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After the translation of the secondary data volume center has been 
made to the coordinate system origin according to the Expressions (16) - (18), then 
there will be the translation of the secondary data volume back to its original position. 
These will provide an error value which now may be used for the remaining automated 
5 alignment steps. 

As discussed, in carrying out the automated alignment portion of the 
autofusion method of the present invention, there most likely will be non-integer align- 
ment requiring interpolation to the optimal alignment position of the secondary data 
volume to the primary data volume. In principle, if the there are two real numbers, for 
10 example, a and b, where a<b, and a third number x s positioned between a and b, such 
that a<x<b. Given this, the value of x based on linear interpolation may be deter- 
mined by Expression (19): 



20 



Thus, linear interpolation in 2-D may be used to resample and reconstruct the second- 
ary volume data in the planes of the primary volume data. In order to effect alignment 
in the 3-D volume data, the 2-D information may be coupled with nearest neighbor 
voxel techniques to obtain the desired interpolation without requiring excessive com- 
putation time, although other interpolation methods may be used. 

Referring to Figure 9, generally at 370, there is a representation of a sit- 
uation in 2-D after transformation of the secondary volume data to the primary coordi- 
nate system. The value of the secondary data at point a, S(a), may be determined by 
interpolation methods such as nearest neighbor, bilinear, B-spline interpolation, or 
higher polynomial methods. In the Figure, the primary volume data is shown at 372 
and the secondary volume data at 374. 

In general, the nearest neighbor method requires that the secondary 
point of interest may be assigned the value of the nearest secondary data point. This 
may be done by rounding the real valued components x, y 9 and z to the corresponding 
nearest integer. Preferably, the present invention uses this interpolation method. 

Before discussing MI, an example of the method used to determine the 
transformations and interpolation, and perform the scan line analysis will be provided. 
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First the two coordinate systems must be related. The alignment of the two 3-D data 
volumes with different x,y,z scale factors implies the determination of the translations 
t X9 t y , t z and the rotations § x , <j> p § z which make the corresponding anatomies coincide. 
Since the complete description of a data volume transformed relative to a fundamental 
5 Cartesian coordinate system may be represented in a matrix of its basis vectors, the 
location of a voxel in one system (the secondary system) transformed to another sys- 
tem (the primary system) may be determined using the systems' basis vectors. There- 
fore, considering the CT image volume (primary) with basis matrix P, origin offset op , 

-> 

jq and basis vectors p*, p y , p z , the basis matrix P is according to Expression (20): 



15 



P = 



P\x Plx P3x °Px 
Ply P2y P3y °Py 
Pit P2z P3z °Pz 

0 0 0 1 



(20) 



The accumulated scaling and rotations are contained in the upper left 3 
x 3 submatrix that is shown in Expression (20). If the origin and axes of the secondary 

volume data coincide with the fundamental Cartesian coordinate system i, j, k of the 
primary volume data, then basis matrix is according to Expression (21): 



25 



P = 



Pix o 0 0 
0 P 2y o 0 
0 0 p 3z 0 
0 0 0 1 



(21) 



30 



In Expression (21), p lx> p 2y p 3z are the lengths of thep x , p>>, pz vectors projected on 

the i, j, k axes. The vector p for the voxel at [rows] [columns] [slices] = [/][m][n] in 
this system has the form shown in Expression (22): 
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P = PP 



Imn 



= p 



= lPlx l + m P2yi + n ^3z k 



(22) 



In Expression (22), the coordinates are shown as integral multiples of the length of the 
basis vectors. 

In the more general system expression at Expression (20), the vector 



p/mn may be shown as Expression (23): 



pimn = P 



Plx Plx P3x °Px 

Ply Ply P3y °Py 

PlzP2z P3z °Pz 
0 0 0 1 



IPlx + m P2x + n P3x + °Px 

lp ly + mp 2y + np 3y + o Py 
lP\ z + mp 2z + np 3z + o Pz 
1 



(23) 



In Expression 23, the components of p ZfWJ on the i, j, k axes are shown in the last term. 

Each of the basis vectors p*, p y , p z have components on all the i, j, k axes thereby 
requiring the use of a more complicated expression than Expression (22). Expression 
(23) provides the physical location of voxel [Z][m][rc] given the integers l,m,n. If matrix 
P has an inverse P" 1 , then the integer location of a voxel may be determined from its 

physical location vector p per Expression (24): 



25 



pZmn 



(24) 



Noting the foregoing in this example, a secondary image volume may 
be represented by a homogeneous basis matrix S as shown in Expression (25): 
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s = 



s lx S 2x S 3x °Sx 



s ly s 2y s 3y °Sy 

S 1 Z S 2z S 3z °Sz 
0 0 0 1 



(25) 



10 



Expression (25) reflects both the origin offset os and basis vectors s x , s y , s z . Like the 
primary system, the equivalence of the physical locations in the secondary system and 
the integer indexed voxels of the secondary volume data are established by the basis 
matrix according to Expressions (26) and (27): 



15 



S — SSlmn 



(26) 



Slmn = S S 



(27) 
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M represents a general homogeneous affine transformation matrix. This 
is shown in Expression (28): 



25 
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M = 



m lx m 2x m 3x *x 

m ly m Zy m 3y t y 
m u m 2z m 3z t z 
0 0 0 1 



(28) 



where, 



m lx m 2x m 3x 

»hy »hy "hy 
m lz m 2 , m 3z j 



=is the upper left 3x3 sub-matrix of M containing scaling and 



rotation information. 



t x , ty, t z = the translation vector. 
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The M matrix of Expression (28) provides that any number of transformation may be 
combined by multiplying matrices with what is shown as m x , nty, m z . 

The transforming of the secondary volume data is equivalent to the 
matrix product shown in Expression (29): 



S' = MS (29) 



10 The location s' of the transformed voxel vector s/mn has the fundamental location that 



is set forth in Expression (30): 



S' = MSs/mn = S's/«n (30) 

15 

In order to display of the intensities of the transformed secondary vol- 
ume data in the primary volume data coordinate system or to compare secondary voxel 
intensities with primary voxel intensities, there must be a determination of the location 

2Q s' in the primary coordinate system. To do this, it is necessary to substitute s for p in 
Expression (24) and the resulting expression is Expression (31): 



J' = P^s' = P" 1 MSs/mn (31) 

25 

Expression (31) will provide the location p' in the primary coordinate system. Thus, 
as the steps of the method of the present invention are performed through the second- 
ary volume data lattice, a corresponding lattice of transformed points is determined in 
3Q the primary coordinate system but at non-integer locations relative to the primary vol- 
ume data lattice. 

The full product of the basis matrices and the transforming matrix in 
Expression (31) is the homogeneous matrix in Expression (32): 
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P _1 MS = X = 



x ll x 21 x 31 X 41 
x 12 x 22 x 32 x 42 
x 13 x 23 x 33 X 43 
x \4 X 24 x 34 X 44 



(32) 



Considering Expression (28) above, the upper left 3x3 sub-matrix of 
this Expression contains the accumulated product of the rotations about the three axes 
and the first three elements of the fourth column contain the components of the transla- 
tions relating the secondary to the primary system origin. In addition, the first the ele- 
ments of the first three columns are the basis vectors for the transformed coordinate 



system relative to the i, j, k axes of the primary coordinate system. The origin offset of 

. ■» "> ■> ■> , . 

the transformed secondary with respect to the primary is o = Ox + o y + o z and shown 
in Figure 10, generally at 400. In Figure 10, the primary volume data is at 402 and the 
secondary volume data is at 404. Since Figure 10 is only a 2-D representation, in the 
x,y plane, it follows that only the x offset at 408 and y offset at 410 are shown. The ori- 
gin offset is obtained from X from Expression (32) according to Expression (33): 



20 



O* = X 41 i, Oy = X 42 j, Oz = * 43 K 



(33) 



The set of the three basis vectors s x , s y , $z is found in Expressions (34), (35), and (36): 



25 



> > > -» 

Sx = x u \ +x 12 j +x 13 k 



(34) 
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Sy = X 2 ]i + X 22 j + x 23^ 



(35) 
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s z = x 31 i +x 32 j +^ 33 k (36) 

5 

•> 

Expressions (34) -(3 6) define the integral stepping away from o along 

the transformed secondary axes s x , Sy as shown in Figure 10. Stepping along the rows, 
columns, and slices away from the origin, any successive secondary voxel location 
may be determined by adding the appropriate vector components. Accordingly, the 
10 four data points a at 412, b at 414, c at 416, and d at 418 will have the locations indi- 
cated below: 

a: lsx + Osy 

b: lsx+lsy 

15 > > 

c: lsx + 2s y 

d: lsje + Ss? 

Thus, the transformed secondary locations are obtained by iterated addition of real- 
time vectors. Further, greater processing time economy may be achieved by rescaling 

20 the Sx, Sy, s z vectors by multiplying by a large integer, such as 2 16 , then changing from 
real to integer data types by truncating the fractional part. The result is that the real 
matrix- vector multiplication has been transformed into integer addition. To rescale the 
data back to the primary coordinate system, the transformed components are divided 
by the same factor which is done by bit shifting. 

25 From the geometry of the primary and the transformed secondary vol- 

ume data, the identity of the voxels that are actually in the field of view on the GUI, or 
that overlap in space are seen and can be processed. In Figure 10, some of the voxels 
on the periphery of the secondary volume data are outside the primary data and would 
not be used in the automated alignment method of the present invention. Only the use- 

30 ful voxels will be transformed, the remainder are not. 

Having now discussed the example with regard to scaling, interpola- 
tion, and transformation, MI maximization will be discussed. 
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Mutual information ("MI") maximization is now used to effect the opti- 
mal registration of the secondary volume data with respect to the primary volume data. 
Initially, there is consideration that primary and secondary volume data A and B, 
respectively, will have voxel values a and b 9 respectively. The voxel values are random 
5 variables that only assume finite integer value. These finite values will lie in the ranges 
of values 0 < a < N A - 1 and 0 < b < N B - 1 , where N A and N B are the numbers of val- 
ues observed in each data, respectively. The probability p(a) that a given A-voxel has 
the value a is defined as the number of voxels with value a, n(a), divided by the num- 
ber of all voxels in A. This is shown in Expression (37): 



15 



25 



, v n (a) 
P(o) = ^ 

^ 07) 



i = o 



The set of all probabilities p(a) form the data set A and will be the probability distribu- 
tion for a. Moreover, it is also understood that for summation of the probabilities for all 
of the values of the voxels in A may be expressed as Expression (38): 

20 jv a -i 

X p(i) = 1 (38) 

i = 0 

Therefore, to determine the p(a) from integer voxel data, a histogram of all the n(a) 
voxel value frequencies is constructed, then normalized according to Expression (37). 

An alignment of the primary and secondary data volumes, such that 
each voxel value a that is part of the primary data A(a e A) has associated with it a 
single secondary voxel b € B , will be based on the probability that the a,b combina- 
tion occurs together according to the joint probability p(a,b) which is set forth in 
30 Expression (39): 
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p{a , b) = (39) 

£ £ n(a, b) 

a e Ab e B 

where, 

n(a,b) = is the number of occurrences of the a,& combination. 

The probability /?(a) of observing a voxel with value a is a measure of 
the uncertainty in predicting this observation. Therefore, for a single random variable 
A with a possible value a e A and a distribution at a given voxel in the primary 
data volume, the entropy (the measure of the uncertainty in the occurrence of a random 
variable) is set forth in Expression (40): 



15 



30 



H(A) = - ]T p{a)\ogp{a) (40) 

a e A 



where, 

A = is a random variable that has the value a. 
lo gp( a)- is a logarithm in base 2. 
20 The entropy, as it is used here, is a function of the distribution p(a) and 

is maximized when the distribution of a is uniform, and is minimized if all of the vox- 
els in A have the same value. Referring to Expression (38), the entropy can be termed 
as a the average of l/logp(a). Therefore, for the two random variables A and B from the 
primary and secondary data volumes, the joint entropy is determined by Expression 
25 (41): 



A B 

H(A, B) = - J £ p(a, b)logp(a, b) (41) 

a = 06 = 0 

where, 

A = is a random variable representing the primary data. 
B = is a random variable representing the secondary data. 
a = is the actual value of random variable A. 
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b = is the actual value of random variable B. 

Given two random variables, one in the primary data volume repre- 
sented by A and one in the secondary data volume represented by B, which have some 
unknown but determinable relationship, that relationship may described as will now be 
5 set forth. The relative entropy of the two random variables A and B is a measure of the 
distance between their two probability distributions. The MI is the amount of informa- 
tion one distribution contains about the other. The relative entropy between the two 
probability distributions, p(a), ae A and p(b), be B , is defined by Expression (42): 



D(p(a)\\p(b)) = £p(a)log|^ (42) 

a g A 

The relative entropy, D, is viewed as a distance measure in that D(p(a) II p(b)) = 0 
15 when p(a) = p{b) , for all a,b, andZ>(p(a) II p(b)) > 0 , for all a,b, 

MI for the purposes of the present invention is the relative entropy 
between the joint distribution and the product of the marginal distribution. Therefore, 
considering Expressions (41) and (42), MI is defined as Expression (43): 



20 



I(A;B)D = XI *> %ffi*" (43) 



a <= Ab e B 



The MI, I(A;B) 9 is the reduction in uncertainty in A due to the knowl- 
edge of B. Therefore, as shown in Figure 2 at graph 158, as the translations and rota- 

2 ^ tions that have been previously described take place automatically, the MI increases 
until it reaches the state shown at 160. After this point, the alignment would only be 
minimally improved with additional computational time. The alignment process 
according to the present invention will normally range from about 30 seconds to 90 
seconds during autofusion. 

3 ^ The terms and expressions that are employed herein are terms or 

description and not of limitation. There is no intention in the use of such terms and 
expressions of excluding the equivalents of the feature shown or described, or portions 
thereof, it being recognized that various modifications are possible within the scope of 
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IN THE CLAIMS : 

1. An autofusion method comprising the steps of: 

(a) generating a first image data volume of a structure using a first modal- 

(b) generating an image data volume of the structure using a second modal- 

(c) processing the first image data volume so that the first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so that the second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plane views of the first 
image data volume at two discrete locations on a display means; 

(f) selectively displaying at least two different plane views of the second 
image data volume at the two discrete locations on the display means, with the plane 
views of the second image data volume being of a same portion of the structure as the 
plane views of the frst image data volume; 

(g) scaling the plane views of the second image data volume to a scale of 
the plane views of the first image data volume; and 

(h) aligning the scaled plane views of the second image data volume with 
the plane views of the first image data volume including the substeps of, 

(1) translating the scaled plane views of the second image data volume 
to a position in each plane view in which predetermined points of the second image 
data volume approximate corresponding points in the first image data volume, 

(2) rotating the scaled and translated plane views of the second image 
data volume to a position in each view in which predetermined points of the second 
image data volume approximate corresponding points in the first image data volume, 
with the approximation at step (h)(2) being a closer approximation than the approxi- 
mation at step (h)(1), and 

(3) substantially simultaneously updating each plane view of the first 
image data volume with each selection movement with respect to the first image data 
volume, and 
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(4) substantially simultaneously updating each plane view of the second 
image data volume with each selection and alignment movement with respect to the 
second data volume. 

2. The method as recited in claim 1, wherein the structure is an anatomy. 

3. The method as recited in claim 2, wherein the first modality is com- 
puted tomography (CT) imaging. 

4. The method as recited in claim 2, wherein the second modality is mag- 
netic resonance imaging tomography imaging (MRI). 

5. The method as recited in claim 1, wherein step (e) includes selectively 
displaying at least three different plane views of the first image data volume at three 
discrete locations on a display means. 

6. The method as recited in claim 5, wherein step (f) includes selectively 
displaying at least three different plane views of the second image data volume at the 
three discrete locations on the display means, with the plane views of the second image 
data volume being of a same portion of the structure as the plane views of the frst 
image data volume. 

7. The method as recited in claim 6, wherein the three different plane 
views of the structure include an axial (transverse) view, coronal view, and sagittal 
view. 

8. An autofusion method comprising the steps of: 

(a) generating a first image data volume of a structure using a first modal- 
ity; 

(b) generating an image data volume of the structure using a second modal- 
ity; 

(c) processing the first image data volume so that the first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so that the second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plane views of the first 
image data volume at two discrete locations on a display means; 

(f) selectively displaying at least two different plane views of the second 
image data volume at the two discrete locations on the display means, with the plane 
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views of the .second image data volume being of a same portion i the structure as the 
plane views or the frst image data volume: 

(g) aligning the scaled plane views of the second ima >; data volume with 
the plane views of the first image data volume including the sub* ops of, 

(1) scaling the plane views of the sceond image d la volume to a scale 
of the plane views of the first image data volume; 

(2) translating the scaled plane views of the secor ' image data volume 
to a position in each plane view in which predetermined points c the second image 
data volume approximate corresponding points in the first image data volume, 

(3) rotating the scaled and translated plane views f the second image 
data volume to a position in each view in which predetermined ] rints of the second 
image data volume approximate corresponding points in the firs image data volume, 
with the approximation at step (g)(3) being a closer approximate m than the approxi- 
mation at step (g)(2), and 

(4) substantially simultaneously updating each p.' ,ne view of the first 
image data volume with each selection movement with respect 1 the first image data 
volume, and 

(5) substantially simultaneously updating each pi nc view of the second 
image data volume with each selection and alignment movemei . with respect lo the 
second data volume. 

9. The method as recited in claim 8, wherein the st icture is an anatomy 

10. The method as recited in claim 9 y wherein the fi -t modality is com- 
puted tomography (CT) imaging* 

1 L The method as recited in claim 9, wherein the s< . ond modality is mag- 
netic resonance imaging tomography imaging (MR1). 

12. The method as recited in claim 8 ? wherein step :) includes selectively 
displaying at least three different plane views of the first image iata volume at three 
discrete locations on a display means. 

1 3. The method as recited in claim 12 7 wherein stej «T) includes selectively 
displaying at least three different plane views of the second itr Jge data volume at the 
three discrete locations on the display means, with the plane vi vs of the second image 
data volume being of a same portion of the structure as the pis \e views of the frst 
image data volume. 
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14. The method as recited in claim 13 T wherein the thi e different plane 
views of the structure include an axial (lransvcr.se) view, coronal iew, and sagittal 
view. 

15. An aulofusion method comprising the steps of: 

(a) generating a first image data volume of a structure using a first modal- 
ity; 

(b) generating an image data volume of the structure } wing a second modal- 

ity; 

(c) processing the first image data volume so that the first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so that .e second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plane dews of the first 
image data volume at two discrete locations on a display means 

(f) selectively displaying at least two different plane "iews of the second 
image data volume at the two discrete locations on the display r ^ans, with the plane 
views of the second image data volume being of a same portion jf the structure as the 
plane views of the first image data volume; 

(g) scaling the second image data volume to a scale if the first image data 
volume; and 

(h) automatically aligning the scaled second image uta volume with the 
first image data volume using mutual information including the :;ubstcps of, 

(1) translating the scaled second image data vok me to a position in 
which each of a plurality of voxels of the second image data v( ame approximate a 
position of corresponding voxels in the first image data volunu 

(2) rotating the scaled and translated second im ije data volume to a 
position in which each of a plurality of voxels or the second in .ige data volume 
approximate a position of corresponding voxels in the first im: ;c data volume, with 
the approximation at step (h)(2) being a closer approximation ian the approximation 
at step (h)(1), and 

(3) substantially simultaneously updating each ilane view of the first 
image data volume with each selection movement with respec to the first image data 
volume, and 
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(4) substantially simultaneously updating each plat * view of the second 
image data volume with each selection and alignment movement >'ith respect to the 
second data volume. 

16. The method as recited in claim 15, wherein the str cture is an anatomy. 

17. The method as recited in claim 16, wherein the fir i modality is com- 
puted tomography (CT) imaging. 

18. The method as recited in claim 1 6, wherein the sec ■ nd modality is mag- 
netic resonance imaging tomography imaging (MRI). 

J 9. The method as recited in claim 15 7 wherein step (e includes selectively 
displaying at least three different plane views of the first image c ta volume at three 
discrete locations on a display means. 

20, The method as recited in claim 19, wherein step (1 includes selectively 
displaying at least three different plane views of the second imag • data volume at the 
three discrete locations on the display means, with the plane viev* i of the second image 
data volume being of a same portion of the structure as the plane views of the first 
image data volume. 

2L The method as recited in claim 20 3 wherein the n ^e different plane 
views of the structure include an axial (transverse) view, corona view, and sagittal 
view. 

22. An autofusion method comprising the steps of: 

(a) generating a first image dala volume of a structu . using a first modal- 



ity; 



ny; 



(b) generating an image data volume of the structure .ising a second modai- 



(c) processing the first image data volume so that tt 1 first image data vol- 
ume is capable of being selectively displayed; 

(d) processing the second image data volume so tha '.he second image data 
volume is capable of being selectively displayed; 

(e) selectively displaying at least two different plan views of the first 
image data volume at two discrete locations on a display mean 

(f) selectively displaying at least two different plar ■ views of the second 
image dala volume at the two discrete locations on the display cieans 7 with the plane 
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views of the second image data volume being of a same portion f the structure as the 
plane views of the first image data volume; 

(g) automatically aligning the scaled plane views of i ic second image data 
volume with the plane views of the first image data volume usin . mutual information 
including the substeps of, 

(1 ) scaling the second image daLa volume to a sc; 3 of the first image 

data volume; 

(2) translating the scaled second image data volu *e to a position in 
which each of the plurality of voxels of the second image data \ ilume approximate a 
position of corresponding voxels in the first image data volume 

(3) rotating the scaled and translated second ima data volume to a 
position in each view in which each of a plurality of voxels of tl : j second image data 
volume approximate a position of corresponding voxels in the Ti it image data volume, 
with the approximation at step (g)(3) being a closer approximat >n than the approxi- 
mation at step (g)(2), and 

(4) substantially simultaneously updating each p :inc view of the first 
image data volume with each selection movement with respect .) the first image data 
volume, and 

(5) substantially simultaneously updating each p ine view of the second 
image data volume with each selection and alignment moveme , with respect to the 
second data volume. 

22. The method as recited in claim 21, wherein the .ructure is an anatomy. 
23- The method as recited in claim 22, wherein the ,rst modality is com- 
puted tomography (CT) imaging. 

24. The method as recited in claim 22, wherein the j cond modality is mag- 
netic resonance imaging tomography imaging (MRI). 

25. The method as recited in claim 22 7 wherein step .e) includes selectively 
displaying at least three different plane views of the first imag data volume at three 
discrete locations on a display means. 

26- The method as recited in claim 25 7 wherein stei (f) includes selectively 
displaying at least three different plane views of the second in .ige data volume at the 
three discrete locations on the display means* with the plane vi vvs of the second image 
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data volume being ol" a same portion of the structure as the plane cws of the first 
image data volume, 

27. The method as recited in claim 26- wherein the tin 2 different plane 
views of the structure include an axial (transverse) view 7 coronal icw, and sagittal 
view. 
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